Date: Fri, 5 Feb 2021 14:47:19 GMT From: Mark Johnston <markj@FreeBSD.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org Subject: git: 3279329b2dca - main - tools/git: Add git-arc Message-ID: <202102051447.115ElJGV080326@gitrepo.freebsd.org>
next in thread | raw e-mail | index | archive | help
The branch main has been updated by markj: URL: https://cgit.FreeBSD.org/src/commit/?id=3279329b2dca12b6f882db7a295b9e790c82f968 commit 3279329b2dca12b6f882db7a295b9e790c82f968 Author: Mark Johnston <markj@FreeBSD.org> AuthorDate: 2021-02-05 14:44:47 +0000 Commit: Mark Johnston <markj@FreeBSD.org> CommitDate: 2021-02-05 14:47:05 +0000 tools/git: Add git-arc This is a handy script for creating and updating Differential revisions from git commits. It tries to avoid forcing the user to manage their git tree in any particular way, but makes two major assumptions: - there is a one-to-one mapping between git commits and Differential revisions, - the title of a Differential revision is the same as the summary line of the corresponding commit. A verbose description of the script's functionality is provided in its usage message, which should probably be converted to a man page. A description of workflows using git-arc is here: https://lists.freebsd.org/pipermail/freebsd-hackers/2021-January/056979.html There are some loose ends but this is functional enough to be useful. Discussed with: jhb Differential Revision: https://reviews.freebsd.org/D28334 --- tools/tools/git/Makefile | 4 + tools/tools/git/git-arc.sh | 596 +++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 600 insertions(+) diff --git a/tools/tools/git/Makefile b/tools/tools/git/Makefile new file mode 100644 index 000000000000..228a41d65275 --- /dev/null +++ b/tools/tools/git/Makefile @@ -0,0 +1,4 @@ +SCRIPTS= git-arc.sh +BINDIR?= /usr/local/bin + +.include <bsd.prog.mk> diff --git a/tools/tools/git/git-arc.sh b/tools/tools/git/git-arc.sh new file mode 100644 index 000000000000..3b85d851319f --- /dev/null +++ b/tools/tools/git/git-arc.sh @@ -0,0 +1,596 @@ +#!/bin/sh +# +# SPDX-License-Identifier: BSD-2-Clause-FreeBSD +# +# Copyright (c) 2019-2021 Mark Johnston <markj@FreeBSD.org> +# Copyright (c) 2021 John Baldwin <jhb@FreeBSD.org> +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions are +# met: +# 1. Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# 2. Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in +# the documentation and/or other materials provided with the distribution. +# +# THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND +# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE +# ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE +# FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS +# OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) +# HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT +# LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY +# OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF +# SUCH DAMAGE. +# + +# TODO: +# - roll back after errors or SIGINT +# - current checkout +# - created revs +# - main (for git arc stage) + +warn() +{ + echo "$(basename $0): $1" >&2 +} + +err() +{ + warn "$1" + exit 1 +} + +err_usage() +{ + cat >&2 <<__EOF__ +Usage: git arc [-vy] <command> <arguments> + +Commands: + create [-l] [-r <reviewer1>[,<reviewer2>...]] [-s subscriber[,...]] [<commit>|<commit range>] + list <commit>|<commit range> + patch <diff1> [<diff2> ...] + stage [-b branch] [<commit>|<commit range>] + update <commit> + +Description: + Create or manage FreeBSD Phabricator reviews based on git commits. There + is a one-to one relationship between git commits and Differential revisions, + and the Differential revision title must match the summary line of the + corresponding commit. In particular, commit summaries must be unique across + all open Differential revisions authored by you. + + The first parameter must be a verb. The available verbs are: + + create -- Create new Differential revisions from the specified commits. + list -- Print the associated Differential revisions for the specified + commits. + patch -- Try to apply a patch from a Differential revision to the + currently checked out tree. + stage -- Prepare a series of commits to be pushed to the upstream FreeBSD + repository. The commits are cherry-picked to a branch (main by + default), review tags are added to the commit log message, and + the log message is opened in an editor for any last-minute + updates. The commits need not have associated Differential + revisions. + update -- Synchronize the Differential revisions associated with the + specified commits. Currently only the diff is updated; the + review description and other metadata is not synchronized. + + The typical end-to-end usage looks something like this: + + $ git commit -m "kern: Rewrite in Rust" + $ git arc create HEAD + <Make changes to the diff based on reviewer feedback.> + $ git commit --amend + $ git arc update HEAD + <Now that all reviewers are happy, it's time to push.> + $ git arc stage HEAD + $ git push freebsd HEAD:main + +Config Variables: + These are manipulated by git-config(1). + + arc.assume_yes [bool] + -- Assume a "yes" answer to all prompts instead of + prompting the user. Equivalent to the -y flag. + + arc.browse [bool] -- Try to open newly created reviews in a browser tab. + Defaults to false. + + arc.list [bool] -- Always use "list mode" (-l) with create. In this + mode, the list of git revisions to create reviews for + is listed with a single prompt before creating + reviews. The diffs for individual commits are not + shown. + + arc.verbose [bool] -- Verbose output. Equivalent to the -v flag. + +Examples: + Create a Phabricator review using the contents of the most recent commit in + your git checkout. The commit title is used as the review title, the commit + log message is used as the review description, markj@FreeBSD.org is added as + a reviewer. + + $ git arc create -r markj HEAD + + Create a series of Phabricator reviews for each of HEAD~2, HEAD~ and HEAD. + Pairs of consecutive commits are linked into a patch stack. Note that the + first commit in the specified range is excluded. + + $ git arc create HEAD~3..HEAD + + Update the review corresponding to commit b409afcfedcdda. The title of the + commit must be the same as it was when the review was created. The review + description is not automatically updated. + + $ git arc update b409afcfedcdda + + Apply the patch in review D12345 to the currently checked-out tree, and stage + it. + + $ git arc patch D12345 + + List the status of reviews for all the commits in the branch "feature": + + $ git arc list main..feature + +__EOF__ + + exit 1 +} + +diff2phid() +{ + local diff + + diff=$1 + if ! expr "$diff" : 'D[1-9][0-9]*$' >/dev/null; then + err "invalid diff ID $diff" + fi + + echo '{"names":["'$diff'"]}' | + arc call-conduit -- phid.lookup | + jq -r "select(.response != []) | .response.${diff}.phid" +} + +diff2status() +{ + local diff tmp status summary + + diff=$1 + if ! expr "$diff" : 'D[1-9][0-9]*$' >/dev/null; then + err "invalid diff ID $diff" + fi + + tmp=$(mktemp) + echo '{"names":["'$diff'"]}' | + arc call-conduit -- phid.lookup > $tmp + status=$(jq -r "select(.response != []) | .response.${diff}.status" < $tmp) + summary=$(jq -r "select(.response != []) | + .response.${diff}.fullName" < $tmp) + printf "%-14s %s\n" "${status}" "${summary}" +} + +log2diff() +{ + local diff + + diff=$(git show -s --format=%B $commit | + sed -nE '/^Differential Revision:[[:space:]]+(https:\/\/reviews.freebsd.org\/)?(D[0-9]+)$/{s//\2/;p;}') + if [ -n "$diff" ] && [ $(echo "$diff" | wc -l) -eq 1 ]; then + echo $diff + else + echo + fi +} + +commit2diff() +{ + local commit diff title + + commit=$1 + + # First, look for a valid differential reference in the commit + # log. + diff=$(log2diff $commit) + if [ -n "$diff" ]; then + echo $diff + return + fi + + # Second, search the open reviews returned by 'arc list' looking + # for a subject match. + title=$(git show -s --format=%s $commit) + diff=$(arc list | fgrep "$title" | egrep -o 'D[1-9][0-9]*:' | tr -d ':') + if [ -z "$diff" ]; then + err "could not find review for '${title}'" + elif [ $(echo "$diff" | wc -l) -ne 1 ]; then + err "found multiple reviews with the same title" + fi + + echo $diff +} + +create_one_review() +{ + local childphid commit dir doprompt msg parent parentphid reviewers + local subscribers + + commit=$1 + reviewers=$2 + subscribers=$3 + parent=$4 + doprompt=$5 + + if [ "$doprompt" ] && ! show_and_prompt $commit; then + return 1 + fi + + git checkout -q $commit + + dir=$(git rev-parse --git-dir)/arc + mkdir -p "$dir" + + msg=${dir}/create-message + git show -s --format='%B' $commit > $msg + printf "\nTest Plan:\n" >> $msg + printf "\nReviewers:\n" >> $msg + printf "${reviewers}\n" >> $msg + printf "\nSubscribers:\n" >> $msg + printf "${subscribers}\n" >> $msg + + yes | env EDITOR=true \ + arc diff --never-apply-patches --create --allow-untracked $BROWSE HEAD~ + [ $? -eq 0 ] || err "could not create Phabricator diff" + + if [ -n "$parent" ]; then + diff=$(commit2diff $commit) + [ -n "$diff" ] || err "failed to look up review ID for $commit" + + childphid=$(diff2phid $diff) + parentphid=$(diff2phid $parent) + echo '{ + "objectIdentifier": "'${childphid}'", + "transactions": [ + { + "type": "parents.add", + "value": ["'${parentphid}'"] + } + ]}' | + arc call-conduit -- differential.revision.edit >&3 + fi + return 0 +} + +# Get a list of reviewers who accepted the specified diff. +diff2reviewers() +{ + local diff phid reviewid userids + + diff=$1 + reviewid=$(diff2phid $diff) + userids=$( \ + echo '{ + "constraints": {"phids": ["'$reviewid'"]}, + "attachments": {"reviewers": true} + }' | + arc call-conduit -- differential.revision.search | + jq '.response.data[0].attachments.reviewers.reviewers[] | select(.status == "accepted").reviewerPHID') + if [ -n "$userids" ]; then + echo '{ + "constraints": {"phids": ['$(echo -n $userids | tr '[:space:]' ',')']} + }' | + arc call-conduit -- user.search | + jq -r '.response.data[].fields.username' + fi +} + +prompt() +{ + local resp + + if [ "$ASSUME_YES" ]; then + return 1 + fi + + printf "\nDoes this look OK? [y/N] " + read resp + + case $resp in + [Yy]) + return 0 + ;; + *) + return 1 + ;; + esac +} + +show_and_prompt() +{ + local commit + + commit=$1 + + git show $commit + prompt +} + +save_head() +{ + local commit orig + + if ! orig=$(git symbolic-ref --short -q HEAD); then + orig=$(git show -s --pretty=%H HEAD) + fi + echo $orig +} + +restore_head() +{ + local orig + + orig=$1 + git checkout -q $orig +} + +build_commit_list() +{ + local chash _commits commits + + for chash in $@; do + _commits=$(git rev-parse "${chash}") + if ! git cat-file -e "${chash}"'^{commit}' >/dev/null 2>&1; then + _commits=$(git rev-list $_commits | tail -r) + fi + [ -n "$_commits" ] || err "invalid commit ID ${chash}" + commits="$commits $_commits" + done + echo $commits +} + +gitarc::create() +{ + local commit commits doprompt list o orig prev reviewers + local subscribers + + list= + if eval $(git config --bool --default false --get arc.list); then + list=1 + fi + doprompt=1 + while getopts lr:s: o; do + case "$o" in + l) + list=1 + ;; + r) + reviewers="$OPTARG" + ;; + s) + subscribers="$OPTARG" + ;; + *) + err_usage + ;; + esac + done + shift $((OPTIND-1)) + + commits=$(build_commit_list $@) + + if [ "$list" ]; then + for commit in ${commits}; do + git --no-pager show --oneline --no-patch $commit + done | git_pager + if ! prompt; then + return + fi + doprompt= + fi + + orig=$(save_head) + prev="" + for commit in ${commits}; do + if create_one_review "$commit" "$reviewers" "$subscribers" "$prev" \ + "$doprompt"; then + prev=$(commit2diff "$commit") + else + prev="" + fi + done + restore_head $orig +} + +gitarc::list() +{ + local chash commit commits diff title + + commits=$(build_commit_list $@) + + for commit in $commits; do + chash=$(git show -s --format='%C(auto)%h' $commit) + echo -n "${chash} " + + diff=$(log2diff $commit) + if [ -n "$diff" ]; then + diff2status $diff + continue + fi + + # This does not use commit2diff as it needs to handle errors + # differently and keep the entire status. The extra 'cat' + # after 'fgrep' avoids erroring due to -e. + title=$(git show -s --format=%s $commit) + diff=$(arc list | fgrep "$title" | cat) + if [ -z "$diff" ]; then + echo "No Review : $title" + elif [ $(echo "$diff" | wc -l) -ne 1 ]; then + echo -n "Ambiguous Reviews: " + echo "$diff" | egrep -o 'D[1-9][0-9]*:' | tr -d ':' \ + | paste -sd ',' - | sed 's/,/, /g' + else + echo "$diff" | sed -e 's/^[^ ]* *//' + fi + done +} + +gitarc::patch() +{ + local rev + + if [ $# -eq 0 ]; then + err_usage + fi + + for rev in $@; do + arc patch --skip-dependencies --nocommit --nobranch --force $rev + echo "Applying ${rev}..." + [ $? -eq 0 ] || break + done +} + +gitarc::stage() +{ + local branch commit commits diff reviewers tmp + + branch=main + while getopts b: o; do + case "$o" in + b) + branch="$OPTARG" + ;; + *) + err_usage + ;; + esac + done + shift $((OPTIND-1)) + + commits=$(build_commit_list $@) + + if [ "$branch" = "main" ]; then + git checkout -q main + else + git checkout -q -b ${branch} main + fi + + tmp=$(mktemp) + for commit in $commits; do + git show -s --format=%B $commit > $tmp + diff=$(arc list | fgrep "$(git show -s --format=%s $commit)" | + egrep -o 'D[1-9][0-9]*:' | tr -d ':') + if [ -n "$diff" ]; then + # XXX this leaves an extra newline in some cases. + reviewers=$(diff2reviewers $diff | sed '/^$/d' | paste -sd ',' - | sed 's/,/, /g') + if [ -n "$reviewers" ]; then + printf "Reviewed by:\t${reviewers}\n" >> $tmp + fi + printf "Differential Revision:\thttps://reviews.freebsd.org/${diff}" >> $tmp + fi + if ! git cherry-pick --no-commit ${commit}; then + warn "Failed to apply $(git rev-parse --short ${commit}). Are you staging patches in the wrong order?" + git checkout -f + break + fi + git commit --edit --file $tmp + done +} + +gitarc::update() +{ + local commit diff orig + + commit=$1 + diff=$(commit2diff $commit) + + if ! show_and_prompt $commit; then + return + fi + + orig=$(save_head) + git checkout -q $commit + + # The linter is stupid and applies patches to the working copy. + # This would be tolerable if it didn't try to correct "misspelled" variable + # names. + arc diff --allow-untracked --never-apply-patches --update $diff HEAD~ + + restore_head $orig +} + +set -e + +ASSUME_YES= +if eval $(git config --bool --default false --get arc.assume-yes); then + ASSUME_YES=1 +fi + +VERBOSE= +while getopts vy o; do + case "$o" in + v) + VERBOSE=1 + ;; + y) + ASSUME_YES=1 + ;; + *) + err_usage + ;; + esac +done +shift $((OPTIND-1)) + +[ $# -ge 1 ] || err_usage + +which arc >/dev/null 2>&1 || err "arc is required, install devel/arcanist" +which jq >/dev/null 2>&1 || err "jq is required, install textproc/jq" + +if [ "$VERBOSE" ]; then + exec 3>&1 +else + exec 3> /dev/null +fi + +case "$1" in +create|list|patch|stage|update) + ;; +*) + err_usage + ;; +esac +verb=$1 +shift + +# All subcommands require at least one parameter. +if [ $# -eq 0 ]; then + err_usage +fi + +# Pull in some git helper functions. +git_sh_setup=$(git --exec-path)/git-sh-setup +[ -f "$git_sh_setup" ] || err "cannot find git-sh-setup" +SUBDIRECTORY_OK=y +USAGE= +. "$git_sh_setup" + +# Bail if the working tree is unclean, except for "list" and "patch" +# operations. +case $verb in +list|patch) + ;; +*) + require_clean_work_tree $verb + ;; +esac + +if eval $(git config --bool --default false --get arc.browse); then + BROWSE=--browse +fi + +gitarc::${verb} $@
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?202102051447.115ElJGV080326>