From owner-freebsd-git@freebsd.org Thu Nov 19 19:00:44 2020 Return-Path: Delivered-To: freebsd-git@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id F2CF3472400 for ; Thu, 19 Nov 2020 19:00:44 +0000 (UTC) (envelope-from dan@langille.org) Received: from wout1-smtp.messagingengine.com (wout1-smtp.messagingengine.com [64.147.123.24]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4CcTXW0wnLz3HDB for ; Thu, 19 Nov 2020 19:00:42 +0000 (UTC) (envelope-from dan@langille.org) Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailout.west.internal (Postfix) with ESMTP id 1F73B76A; Thu, 19 Nov 2020 14:00:38 -0500 (EST) Received: from mailfrontend1 ([10.202.2.162]) by compute4.internal (MEProxy); Thu, 19 Nov 2020 14:00:38 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=langille.org; h= content-type:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; s=fm2; bh=/ 37srXCWvBumg81HJaaLDO62IrhCpF8Klo6wrSKX9YA=; b=SMcaRRmtb/NO5M4vU lq2GFCgY9RSKr0pPsWt2sVYc0Vx0uI0TJu7xKqwmJa8QoYLWuxT4XdM3W6hzHbW5 /A5O8TSDqhaJpZKQL7mlYZIWqfHab12ROgEx1fq86S+j4tGewzep2XjhZWGhUXlX cD9ProkghILUYjCMhPE3AsGlM7WOT8BtKJ4Z+wa0qzP+hxrJw9KL6TYReZJr/UEy C4AC+qzkEW5L6NGB9/xLVB+Kcu8S/cM5nB/reBglQ6v/puS5r5WyV64NhvK0syng x2yHtujCFvvDBPjCaA12UJVpEPkW/zKisCXfUXdtvp/UdMb8x1T6NC86ecwrwLAF zJikQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:content-type :date:from:in-reply-to:message-id:mime-version:references :subject:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm1; bh=/37srXCWvBumg81HJaaLDO62IrhCpF8Klo6wrSKX9 YA=; b=pfSSuTS0odhuf85glEyiMnXbSTFYpNA/PiCu5mWUwPZEOzX3xiM1hGZvk 2p+nyYRE+NJJC/qqCslmBScu/aFVp3/3cfPsGrHE9Yy7jE5f2QA6rLc8eaxPNp9J oSroZShWEC4LZbypcVXgOclC0420VLVM26IotGjFDtWfc1HKxBoS85ABbZ08OmRV WuUW4Zihfskywr54NxbmtUqRU/QEYY59GcBGimslxs2tv2JpWLDK0JRghzCP0GTR Lhrbv8SevETh2DX6q67Xo+gpSpGLmteXbLn+qzkZPtzJYBRlxfNqwYthv4fpTOZW O9ZvRcsbwUS03CwHNOvpMiU6EZAuA== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedujedrudefjedguddukecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfgh necuuegrihhlohhuthemuceftddtnecunecujfgurheptggguffhjgffgffkfhfvofesth hqmhdthhdtjeenucfhrhhomhepffgrnhcunfgrnhhgihhllhgvuceouggrnheslhgrnhhg ihhllhgvrdhorhhgqeenucggtffrrghtthgvrhhnpeeggfegfeeukeeigfeiudektefhte ffledugeekfeehjeevteeifeeigefgjeevieenucffohhmrghinheplhgrnhhgihhllhgv rdhorhhgnecukfhppedutdekrdehvddrvdefjedrudeffeenucevlhhushhtvghrufhiii gvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpegurghnsehlrghnghhilhhlvgdrohhr gh X-ME-Proxy: Received: from pro02.wifi.int.unixathome.org (pool-108-52-237-133.phlapa.fios.verizon.net [108.52.237.133]) by mail.messagingengine.com (Postfix) with ESMTPA id F0CFD3280065; Thu, 19 Nov 2020 14:00:36 -0500 (EST) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.120.23.2.4\)) Subject: Re: Monitoring commits on all branches From: Dan Langille In-Reply-To: <3c9f6285-ae7c-1062-2dd3-42f8c953a230@gmail.com> Date: Thu, 19 Nov 2020 14:00:36 -0500 Cc: freebsd-git@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: References: <197541CC-FEA7-4B4C-936E-66A5625BB64C@langille.org> <3c9f6285-ae7c-1062-2dd3-42f8c953a230@gmail.com> To: Marc Branchaud X-Mailer: Apple Mail (2.3608.120.23.2.4) X-Rspamd-Queue-Id: 4CcTXW0wnLz3HDB X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=langille.org header.s=fm2 header.b=SMcaRRmt; dkim=pass header.d=messagingengine.com header.s=fm1 header.b=pfSSuTS0; dmarc=pass (policy=none) header.from=langille.org; spf=pass (mx1.freebsd.org: domain of dan@langille.org designates 64.147.123.24 as permitted sender) smtp.mailfrom=dan@langille.org X-Spamd-Result: default: False [-3.60 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[]; MV_CASE(0.50)[]; R_SPF_ALLOW(-0.20)[+ip4:64.147.123.24]; RCVD_COUNT_THREE(0.00)[4]; DKIM_TRACE(0.00)[langille.org:+,messagingengine.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[langille.org,none]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FREEMAIL_TO(0.00)[gmail.com]; RECEIVED_SPAMHAUS_PBL(0.00)[108.52.237.133:received]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:11403, ipnet:64.147.123.0/24, country:US]; RCVD_TLS_LAST(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_IN_DNSWL_LOW(-0.10)[64.147.123.24:from]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[langille.org:s=fm2,messagingengine.com:s=fm1]; FREEFALL_USER(0.00)[dan]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RWL_MAILSPIKE_VERYGOOD(0.00)[64.147.123.24:from]; MAILMAN_DEST(0.00)[freebsd-git] X-BeenThere: freebsd-git@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Discussion of git use in the FreeBSD project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 Nov 2020 19:00:45 -0000 > On Nov 19, 2020, at 11:16 AM, Marc Branchaud = wrote: >=20 > On 2020-11-18 8:49 p.m., Dan Langille wrote: >> How can a repo be monitored for commits on all branches? >> I know how to ask a given branch: do you have any commits after = foo_hash? >> How do I: >> * get a list of all commits since foo_hash >=20 > A quick a note about Warner's reply: >=20 >> git log $hash..HEAD >=20 > "HEAD" is just a git nickname for "whatever you have currently = checked-out" (which can be a branch, a tag, or "detached" commit SHA = ID). Mathieu mentioned: git log $foo_hash...branch_name That was the first time I've seen that used. All previous suggestions = were HEAD, not branch_name. But they are all the same in this context? >> * know which branch each of those commits was on (e.g. master, = branches/2020Q4) >=20 > Unfortunately you'll find most normal git advice to be a bit = frustrating with the FreeBSD repos, because FreeBSD doesn't work the way = most people use git. Specifically, the FreeBSD project does not ever = merge branches (in the git sense of the word "merge"). Things would be = very, very much easier if the FreeBSD project were to use git-style = merging. I believe there are discussions underway about adjusting the = whole MFC process for the git world. I admit that part of my motivation = in writing this message is to provide grist for that mill. >=20 > Fortunately even without git-merged branches, there are still git = tools that help, though they're not as precise as one would like. >=20 > Let's look at a concrete example with the beta ports git repo (which I = just cloned), and compare the 2020Q4 and main branches. I'll start with = some overall exploration, then address your specific question. >=20 > There are 298 commits in the 2020Q4 branch. I know this because > git merge-base origin/main origin/branches/2020Q4 > tells me where 2020Q4 branched off of main: commit 5dbe4e5f775ea2. = And > git rev-list 5dbe4e5f775ea2..origin/branches/2020Q4 | wc -l > says "299". (The "rev-list" command is a bare-bones version of "log" = that only lists commit SHA IDs.) [examples snipped] I followed that. I took the merge information as background and good-to-know, because = FreshPorts won't be doing any merging. It just needs a good "git = checkout" working copy. Sorry for such a long reply. * How FreshPorts extracts data FreshPorts is only interested in a snapshot of the repo with respect to = a given commit. It works on the 'repo as a whole' to extract values = from the ports which were affected by that commit. Case in point: a = commit to a parent port might affect any or all of the child ports. All = the child ports need to be refreshed. I am quickly concluding that FreshPorts must decide in advance what git = branches it will pay attention to. At present, it follows all branches. * FreshPorts (without git) uses email to create XML When moving FreshPorts from subversion to git, one of the goals was to = avoid relying on email to know that a commit has occurred. That is how = FreshPorts has always worked. The email (from the CVS commit) was parsed = and XML created. This code was updated for SVN. The XML is then used to = load the commit into the FreshPorts database which then drives the = website contents. When I started the GIT conversion, there was no commit email. "git log = $foo_hash...HEAD" is how FreshPorts knows what commits to process. One positive aspect of email approach: it identified the branch. So far, = I can't see how I can process the repo as a whole and see every commit = and know what branch it was on. * Polling git It is beginning to sound like the FreshPorts git code for detecting = incoming commits will be: Every N minutes, do this: for each repo in REPOS for branch in BRANCHES cd to the directory for that repo git checkout branch git log $branch_last_hash...HEAD for each of those commits process the commit end for end for end for At present, the REPOS and BRANCHES are: * freebsd BRANCHES=3D"master branches/2020Q4 branches/2020Q3 = branches/2020Q2 ...etc" * freebsd-ports BRANCHES=3D"master stable/12 stable/11" * freebsd-doc BRANCHES=3D"master" Some might ask: * Why not just master and latest-quarterly for freebsd-ports? * Because commits to older branches sometimes occur (or at least I = thought I saw one once) Commit hooks might also help, but I'm not sure if that will make things = easier or complicate everything * When new branches arrive It is vital that FreshPorts remain automated as much as it can be. At = present, under SVN, I might have to fix things perhaps 5 or 6 times a = year, usually because a commit did not get processed. Keeping that in mind, I do not yet know how to handle the following = situations: A new branch comes out. * Automation might be possible for ports quarterly branches * FreshPorts has to know there is a new branch * BRANCHES needs to be updated * I don't see that it can be automated for stable/* * need to handle 'git checkout branch' when branch does not exist? * Once branch exists, how do you find out about the commits when you = have no starting point for 'git log'? Right now, a new quarterly branch is noticed when the first commit email = comes through. FreshPorts then does an 'svn co' for that branch. I'm hoping someone has good ideas for my edge cases. Thank you. =E2=80=94=20 Dan Langille http://langille.org/