From owner-svn-src-user@freebsd.org Wed Jan 4 17:55:33 2017 Return-Path: Delivered-To: svn-src-user@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 46E90C9F074 for ; Wed, 4 Jan 2017 17:55:33 +0000 (UTC) (envelope-from uqs@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 219F116C3; Wed, 4 Jan 2017 17:55:33 +0000 (UTC) (envelope-from uqs@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id v04HtWc7014301; Wed, 4 Jan 2017 17:55:32 GMT (envelope-from uqs@FreeBSD.org) Received: (from uqs@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id v04HtVLW014295; Wed, 4 Jan 2017 17:55:31 GMT (envelope-from uqs@FreeBSD.org) Message-Id: <201701041755.v04HtVLW014295@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: uqs set sender to uqs@FreeBSD.org using -f From: Ulrich Spoerlein Date: Wed, 4 Jan 2017 17:55:31 +0000 (UTC) To: src-committers@freebsd.org, svn-src-user@freebsd.org Subject: svn commit: r311295 - in user/uqs/git_conv: base-head.git/.git doc-head.git/.git freebsd-base.git freebsd-doc.git freebsd-ports.git ports-head.git/.git X-SVN-Group: user MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-user@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "SVN commit messages for the experimental " user" src tree" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 04 Jan 2017 17:55:33 -0000 Author: uqs Date: Wed Jan 4 17:55:31 2017 New Revision: 311295 URL: https://svnweb.freebsd.org/changeset/base/311295 Log: Bring back the fetch configs again, while unused on the repo converter, it's a handy way to check your own conversion against what is published on github. Remove the 'www' branch for the doc repo, which is old and wasn't really used anyway. Modified: user/uqs/git_conv/base-head.git/.git/config user/uqs/git_conv/doc-head.git/.git/config user/uqs/git_conv/freebsd-base.git/config user/uqs/git_conv/freebsd-doc.git/config user/uqs/git_conv/freebsd-ports.git/config user/uqs/git_conv/ports-head.git/.git/config Modified: user/uqs/git_conv/base-head.git/.git/config ============================================================================== --- user/uqs/git_conv/base-head.git/.git/config Wed Jan 4 17:54:44 2017 (r311294) +++ user/uqs/git_conv/base-head.git/.git/config Wed Jan 4 17:55:31 2017 (r311295) @@ -10,5 +10,6 @@ [push] default = matching [remote "github"] - url = src.github.com:freebsd/freebsd.git + url = base.github.com:freebsd/freebsd.git + fetch = +refs/heads/*:refs/remotes/github/* push = +refs/remotes/trunk:refs/heads/svn_head Modified: user/uqs/git_conv/doc-head.git/.git/config ============================================================================== --- user/uqs/git_conv/doc-head.git/.git/config Wed Jan 4 17:54:44 2017 (r311294) +++ user/uqs/git_conv/doc-head.git/.git/config Wed Jan 4 17:55:31 2017 (r311295) @@ -7,10 +7,9 @@ rewriteRoot = svn+ssh://svn.freebsd.org/doc url = file:///home/svn/doc fetch = head:refs/remotes/trunk - branches = {www}:refs/remotes/* [push] default = matching [remote "github"] url = doc.github.com:freebsd/freebsd-doc.git + fetch = +refs/heads/*:refs/remotes/github/* push = +refs/remotes/trunk:refs/heads/svn_head - push = +refs/remotes/www:refs/heads/svn_www Modified: user/uqs/git_conv/freebsd-base.git/config ============================================================================== --- user/uqs/git_conv/freebsd-base.git/config Wed Jan 4 17:54:44 2017 (r311294) +++ user/uqs/git_conv/freebsd-base.git/config Wed Jan 4 17:55:31 2017 (r311295) @@ -6,6 +6,7 @@ default = matching [remote "github"] url = base.github.com:freebsd/freebsd.git + fetch = +refs/heads/*:refs/remotes/github/* push = +refs/heads/master:refs/heads/master push = +refs/heads/projects/*:refs/heads/projects/* push = +refs/heads/release/*:refs/heads/release/* Modified: user/uqs/git_conv/freebsd-doc.git/config ============================================================================== --- user/uqs/git_conv/freebsd-doc.git/config Wed Jan 4 17:54:44 2017 (r311294) +++ user/uqs/git_conv/freebsd-doc.git/config Wed Jan 4 17:55:31 2017 (r311295) @@ -6,6 +6,7 @@ default = matching [remote "github"] url = doc.github.com:freebsd/freebsd-doc.git + fetch = +refs/heads/*:refs/remotes/github/* push = +refs/heads/master:refs/heads/master push = +refs/heads/projects/*:refs/heads/projects/* push = +refs/heads/release/*:refs/heads/release/* Modified: user/uqs/git_conv/freebsd-ports.git/config ============================================================================== --- user/uqs/git_conv/freebsd-ports.git/config Wed Jan 4 17:54:44 2017 (r311294) +++ user/uqs/git_conv/freebsd-ports.git/config Wed Jan 4 17:55:31 2017 (r311295) @@ -6,6 +6,7 @@ default = matching [remote "github"] url = ports.github.com:freebsd/freebsd-ports.git + fetch = +refs/heads/*:refs/remotes/github/* push = +refs/heads/branches/*:refs/heads/branches/* push = +refs/heads/master:refs/heads/master push = +refs/heads/projects/*:refs/heads/projects/* Modified: user/uqs/git_conv/ports-head.git/.git/config ============================================================================== --- user/uqs/git_conv/ports-head.git/.git/config Wed Jan 4 17:54:44 2017 (r311294) +++ user/uqs/git_conv/ports-head.git/.git/config Wed Jan 4 17:55:31 2017 (r311295) @@ -11,4 +11,5 @@ default = matching [remote "github"] url = ports.github.com:freebsd/freebsd-ports.git + fetch = +refs/heads/*:refs/remotes/github/* push = +refs/remotes/trunk:refs/heads/svn_head From owner-svn-src-user@freebsd.org Wed Jan 4 17:58:18 2017 Return-Path: Delivered-To: svn-src-user@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 78232C9F0DA for ; Wed, 4 Jan 2017 17:58:18 +0000 (UTC) (envelope-from uqs@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 52192184D; Wed, 4 Jan 2017 17:58:18 +0000 (UTC) (envelope-from uqs@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id v04HwHbu014439; Wed, 4 Jan 2017 17:58:17 GMT (envelope-from uqs@FreeBSD.org) Received: (from uqs@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id v04HwHkC014438; Wed, 4 Jan 2017 17:58:17 GMT (envelope-from uqs@FreeBSD.org) Message-Id: <201701041758.v04HwHkC014438@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: uqs set sender to uqs@FreeBSD.org using -f From: Ulrich Spoerlein Date: Wed, 4 Jan 2017 17:58:17 +0000 (UTC) To: src-committers@freebsd.org, svn-src-user@freebsd.org Subject: svn commit: r311296 - user/uqs/git_conv X-SVN-Group: user MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-user@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "SVN commit messages for the experimental " user" src tree" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 04 Jan 2017 17:58:18 -0000 Author: uqs Date: Wed Jan 4 17:58:17 2017 New Revision: 311296 URL: https://svnweb.freebsd.org/changeset/base/311296 Log: Snapshot what is currently in production. Finish the implicit "src" removal and use base/doc/ports more consistenly. Add more documentation and caveats to people that want to repeat this setup. Modified: user/uqs/git_conv/git_conv Modified: user/uqs/git_conv/git_conv ============================================================================== --- user/uqs/git_conv/git_conv Wed Jan 4 17:55:31 2017 (r311295) +++ user/uqs/git_conv/git_conv Wed Jan 4 17:58:17 2017 (r311296) @@ -1,38 +1,75 @@ #!/bin/sh # -# Repository creation and setup +# 1. Repository creation and setup # -# Simple for svn2git repos, need to run against local paths, using rules files in ~git/*.rules -# - First svn2git run will create target git repo, then +# svnsync mirror with cloned UUID +# - svnadmin create freebsd-base.svn +# - svnadmin create freebsd-doc.svn +# - svnadmin create freebsd-ports.svn +# - echo '#!/bin/sh' > freebsd-base.svn/hooks/pre-revprop-change +# - echo '#!/bin/sh' > freebsd-doc.svn/hooks/pre-revprop-change +# - echo '#!/bin/sh' > freebsd-ports.svn/hooks/pre-revprop-change +# - chmod +x freebsd-base.svn/hooks/pre-revprop-change freebsd-doc.svn/hooks/pre-revprop-change freebsd-ports.svn/hooks/pre-revprop-change +# - svnadmin setuuid freebsd-base.svn ccf9f872-aa2e-dd11-9fc8-001c23d0bc1f +# - svnadmin setuuid freebsd-doc.svn c2e8774f-c49f-e111-b436-862b2bbc8956 +# - svnadmin setuuid freebsd-ports.svn 35697150-7ecd-e111-bb59-0022644237b5 +# - svnsync init file://$PWD/freebsd-base.svn svn://svn.freebsd.org/base +# - svnsync init file://$PWD/freebsd-doc.svn svn://svn.freebsd.org/doc +# - svnsync init file://$PWD/freebsd-ports.svn svn://svn.freebsd.org/ports +# !!! ACHTUNG !!! +# svnsync doesn't work atomically and can mess up metadata for commits. This +# has happened already with the mirror as used by the official svn2git +# conversion as exported in github. This is even worse than you can imagine. +# +# DO NOT USE THIS NAIVE SVN SYNC FROM THE LIVE REPO, see dosync.sh instead. +# - svnsync sync file://$PWD/freebsd-base.svn +# - svnsync sync file://$PWD/freebsd-doc.svn +# - svnsync sync file://$PWD/freebsd-ports.svn +# this will take several days, possibly weeks. +# !!! ACHTUNG !!! +# +# 2. svn2git conversion: need to run against local paths, using rules files in ~git/*.rules +# - First svn2git run will create the target git repo, then # - git config --global push.default matching -# - git remote add github github.com:freebsd/freebsd.git +# - git remote add github base.github.com:freebsd/freebsd.git # - git config --add remote.github.push '+refs/heads/master:refs/heads/master' # - git config --add remote.github.push '+refs/heads/stable/*:refs/heads/stable/*' # - git config --add remote.github.push '+refs/heads/projects/*:refs/heads/projects/*' # - git config --add remote.github.push '+refs/notes/*:refs/notes/*' # - etc. # - touch freebsd-base.git/git-daemon-export-ok -# - svn2git/svn-all-fast-export --rules ~git/freebsd-base.rules --add-metadata-notes --identity-domain FreeBSD.org /home/svn/base +# - svn2git/svn-all-fast-export --rules freebsd-base.rules --add-metadata-notes --identity-domain FreeBSD.org /home/svn/freebsd-base.svn # - git push github # done. Analog steps needed for doc and ports. # -# For git-svn it's a bit more involved. We use separate working dirs, but push to the same destination repo. -# - git svn init -Thead -rewrite-root=svn+ssh://svn.freebsd.org/base file:///home/svn/base src-head.git -# - cd src-head.git +# 3. git-svn conversion: we use separate working dirs, but push to the same destination repo. +# - git svn init -Thead --rewrite-root=svn+ssh://svn.freebsd.org/base file:///home/svn/freebsd-base.svn base-head.git +# - git svn init -Thead --rewrite-root=svn+ssh://svn.freebsd.org/doc file:///home/svn/freebsd-doc.svn doc-head.git +# - git svn init -Thead --rewrite-root=svn+ssh://svn.freebsd.org/ports file:///home/svn/freebsd-ports.svn ports-head.git +# - cd base-head.git # - git svn fetch -r 0:1000 # - git config --global push.default matching -# - git remote add github github.com:freebsd/freebsd.git +# - git remote add github base.github.com:freebsd/freebsd.git # - git config --add remote.github.push '+refs/remotes/trunk:refs/heads/svn_head' # - git svn rebase # - git push github - +# +# 4. For using different SSH keys with github, you need to fuzz your .ssh/config like so: +# +# Host base.github.com +# User git +# Hostname github.com +# IdentityFile ~/.ssh/github_base +# +# And similar for ports and doc. : ${BASE=/home/git} : ${RULES_DIR=${BASE}} : ${SVN2GIT=/usr/local/bin/svn2git} -: ${SRC_REPO=/home/svn/base} -: ${DOC_REPO=/home/svn/doc} -: ${PORTS_REPO=/home/svn/ports} +# xxx_REPO are legacy names, TODO remove them ... +: ${REPO_base=${SRC_REPO:-/home/svn/base}} +: ${REPO_doc=${DOC_REPO:-/home/svn/doc}} +: ${REPO_ports=${PORTS_REPO:-/home/svn/ports}} usage() { @@ -98,20 +135,59 @@ svn2git() cd $BASE echo "Converting $source to $target using svn2git" - $SVN2GIT --add-metadata-notes --identity-domain FreeBSD.org \ - --rules $rules $source + $SVN2GIT --add-metadata-notes --identity-domain FreeBSD.org --rules $rules $source if [ $? != 0 ]; then echo "Error in svn2git conversion of $source" >&2 exit 1 fi touch $target/git-daemon-export-ok + cd $target + if ! git config --local --get push.default >/dev/null; then + git config --local push.default matching + fi + if ! git config --local --get remote.github.url >/dev/null; then + # We use fake github hosts like ports.github.com, which we fix in our + # .ssh/config *and* apply a per-repo ssh key, which is the whole point + # of this hack. + case "$TYPE" in + base) + git remote add github $TYPE.github.com:freebsd/freebsd.git + git config --add remote.github.push '+refs/heads/master:refs/heads/master' + git config --add remote.github.push '+refs/heads/projects/*:refs/heads/projects/*' + git config --add remote.github.push '+refs/heads/release/*:refs/heads/release/*' + git config --add remote.github.push '+refs/heads/releng/*:refs/heads/releng/*' + git config --add remote.github.push '+refs/heads/stable/*:refs/heads/stable/*' + git config --add remote.github.push '+refs/heads/user/*:refs/heads/user/*' + git config --add remote.github.push '+refs/notes/*:refs/notes/*' + + git remote add bitbucket ssh://git@bitbucket.org/freebsd/freebsd-$TYPE.git + git config --add remote.bitbucket.push '+refs/heads/master:refs/heads/master' + git config --add remote.bitbucket.push '+refs/heads/stable/*:refs/heads/stable/*' + git config --add remote.bitbucket.push '+refs/notes/*:refs/notes/*' + ;; + doc) + git remote add github $TYPE.github.com:freebsd/freebsd-$TYPE.git + git config --add remote.github.push '+refs/heads/master:refs/heads/master' + git config --add remote.github.push '+refs/heads/projects/*:refs/heads/projects/*' + git config --add remote.github.push '+refs/heads/release/*:refs/heads/release/*' + git config --add remote.github.push '+refs/notes/*:refs/notes/*' + ;; + ports) + git remote add github $TYPE.github.com:freebsd/freebsd-$TYPE.git + git config --add remote.github.push '+refs/heads/branches/*:refs/heads/branches/*' + git config --add remote.github.push '+refs/heads/master:refs/heads/master' + git config --add remote.github.push '+refs/heads/projects/*:refs/heads/projects/*' + git config --add remote.github.push '+refs/notes/*:refs/notes/*' + ;; + esac + fi if [ -z "$dest" -o -n "$run_local" ]; then return fi echo "Pushing $target to $dest" - cd $target && for d in $dest; do + for d in $dest; do git push $d || { echo "Error in pushing to $dest" >&2; exit 1; } done } @@ -122,7 +198,18 @@ gitsvn() target=$1; shift dest="$@" - test -d "$BASE/$target/.git" || { echo "$BASE/$target is not a git repo, exiting ..." >&2; exit 1; } + cd $BASE + if ! test -d "$BASE/$target/.git"; then + echo "$BASE/$target is not a git repo, creating first revisions ..." + git svn init -Thead --rewrite-root=svn+ssh://svn.freebsd.org/$TYPE file://`eval echo \\\${REPO_${TYPE}}` $BASE/$target + cd $BASE/$target && git svn fetch -r 0:100 + git config --local push.default matching + case "$TYPE" in + base) git remote add github $TYPE.github.com:freebsd/freebsd.git ;; + *) git remote add github $TYPE.github.com:freebsd/freebsd-$TYPE.git ;; + esac + git config --add remote.github.push '+refs/remotes/trunk:refs/heads/svn_head' + fi cd $BASE/$target echo "Converting $target using git-svn" git svn rebase @@ -143,7 +230,7 @@ gitsvn() if [ -n "$do_gitsvn" ]; then case "$TYPE" in - base) gitsvn src-head.git github ;; + base) gitsvn base-head.git github ;; doc) gitsvn doc-head.git github ;; ports) gitsvn ports-head.git github ;; esac @@ -151,8 +238,8 @@ fi if [ -n "$do_svn2git" ]; then case "$TYPE" in - base) svn2git $RULES_DIR/freebsd-base.rules ${SRC_REPO} github bitbucket ;; - doc) svn2git $RULES_DIR/freebsd-doc.rules ${DOC_REPO} github ;; - ports) svn2git $RULES_DIR/freebsd-ports.rules ${PORTS_REPO} github ;; + base) svn2git $RULES_DIR/freebsd-base.rules ${REPO_base} github bitbucket ;; + doc) svn2git $RULES_DIR/freebsd-doc.rules ${REPO_doc} github ;; + ports) svn2git $RULES_DIR/freebsd-ports.rules ${REPO_ports} github ;; esac fi