Date: Sat, 3 Dec 2016 13:40:53 -0500 From: Shawn Webb <shawn.webb@hardenedbsd.org> To: Ulrich Sp??rlein <uqs@freebsd.org> Cc: freebsd-git@freebsd.org Subject: Re: Git mirroring halted for freebsd-base Message-ID: <20161203184053.GE82677@mutt-hardenedbsd> In-Reply-To: <CAJ9axoRNMaBD6xfnVMbpdQZkB6Gc0nqYGuiMGBEmE2f%2B0r7G5A@mail.gmail.com> References: <CAJ9axoQH0FuaRghpc3OVTgxT42ykObTni3m9EgHsDKnKFfH7dA@mail.gmail.com> <CAJ9axoR5RFFpX3eTKxg0piLn4UidNrW9jCm-gFJS=m0xVOumzA@mail.gmail.com> <20161203172151.GA82677@mutt-hardenedbsd> <CAJ9axoSthwAEn9R-b=LdnOGL%2BoXagp8jcU_RfykxRebqLh6SYA@mail.gmail.com> <20161203172910.GB82677@mutt-hardenedbsd> <CAJ9axoRRN%2B0H2Aptipe4mGFwOjSnbq1-xVB5%2B8SF0e2%2BNC7-1w@mail.gmail.com> <20161203181233.GC82677@mutt-hardenedbsd> <CAJ9axoRNMaBD6xfnVMbpdQZkB6Gc0nqYGuiMGBEmE2f%2B0r7G5A@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--3O1VwFp74L81IIeR Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sat, Dec 03, 2016 at 07:16:08PM +0100, Ulrich Sp??rlein wrote: > 2016-12-03 19:12 GMT+01:00 Shawn Webb <shawn.webb@hardenedbsd.org>: > > On Sat, Dec 03, 2016 at 06:45:32PM +0100, Ulrich Sp??rlein wrote: > >> 2016-12-03 18:29 GMT+01:00 Shawn Webb <shawn.webb@hardenedbsd.org>: > >> > On Sat, Dec 03, 2016 at 06:25:29PM +0100, Ulrich Sp??rlein wrote: > >> >> 2016-12-03 18:21 GMT+01:00 Shawn Webb <shawn.webb@hardenedbsd.org>: > >> >> > On Sat, Dec 03, 2016 at 12:42:56PM +0100, Ulrich Sp??rlein wrote: > >> >> >> 2016-12-02 11:07 GMT+01:00 Ulrich Sp??rlein <uqs@freebsd.org>: > >> >> >> > The conversion process started chewing up 100% cpu without mak= ing much progress, first attempts to rectify this have failed. > >> >> >> > > >> >> >> > The svn2git conversion and pushes to github have been halted. = Pushes to bitbucket also have been halted (we're reaching the 2GB limit imp= osed by bitbucket). > >> >> >> > > >> >> >> > I'll update this thread in about 24h. > >> >> >> > Uli > >> >> >> > >> >> >> Service is fully restored now for the github mirror, sorry for t= he downtime. > >> >> >> Bitbucket will stop working soon because of the size limitations. > >> >> >> Speak up if you require this mirror to be kept up-to-date. > >> >> >> > >> >> >> Cheers, > >> >> >> Uli > >> >> > > >> >> > Looks like it might be easier for some downstream projects to ful= ly > >> >> > recreate their ports repositories from scratch than to try to mer= ge from > >> >> > upstream. > >> >> > >> >> What are you referring to here? > >> > > >> > The ports repo at https://github.com/freebsd/freebsd-ports was force > >> > pushed. Now attempts at merging in upstream's ports tree into > >> > hardenedbsd's causes merge conflicts for hundreds of files, including > >> > files we didn't change. > >> > > >> > So I'm forced to either inspect hundreds of files, manually merging = in > >> > the changes or recreate our ports tree from scratch, re-importing our > >> > changes in a single atomic commit. The second option sounds more > >> > appealing, though we'd lose the entire history of our changes. > >> > > >> > Additionally, anyone downstream from HardenedBSD might have to do the > >> > same. Domino affect. > >> > >> I see. This shouldn't have happend, but as svnsync is > >> non-transactional, we picked up some bad SVN metadata that made it > >> into ports and base repos about a year ago. The SVN corruption was > >> promptly fixed (I didn't ask for this), but that now leaves us with no > >> way to actually re-do the conversion from scratch, as you'd need a > >> corrupted SVN repo to produce the same results. > >> > >> You should be able to simply merge whatever "official" commit you last > >> merged to with whatever the new "official" commit is now. This only > >> affected metadata, so you'll get a clean merge (no conflicts) but you > >> end up depending on 2x the history for about a year or so. Shouldn't > >> be that much of a problem. Ask your local git wizard on how to do this > >> best. > >> > >> >> > What caused the issue? What is going to be done to prevent it from > >> >> > happening again? > >> >> > >> >> I have no root cause, other than bitbucket changing permissions and > >> >> somehow git ending up using 100% CPU for most of the operations. > >> > > >> > So no guarantees this massive screw-up won't happen again? > >> > >> I said this before, and I'll say it again. This is a best-effort > >> conversion and we're at the mercy of whatever SVN fucks up next. I > >> provided clear instructions as to how to do the conversion in-house, > >> and guess how many people actually wrote to me that they end up with > >> different SHA hashes on github than they can produce in-house for both > >> src and ports? > >> > >> What would be your guess? > >> > >> Exactly, 0 people have done the in-house conversion and have compared > >> this to github. I could have put all kinds of backdoors in FreeBSD on > >> github and not a single soul would've noticed. > >> > >> So if you depend on it, I would very much appreciate if you could do > >> the same conversion in-house and report any drift as soon as possible, > >> because it's a mess otherwise, as you can see. > >> > >> Any thoughts on how to fix this for src would also be appreciated, all > >> I can think of is either pushing 2 heads and telling people to > >> migrate, or doing the switchover on a flag day. > > > > Hey Uli, > > > > Sorry for the harsher tone earlier. I'm a bit stressed and it was unfair > > of me to use that tone. > > > > I'm grateful for your efforts. I understand that supporting git isn't an > > official service provided by FreeBSD. One item on my Christmas wishlst > > would be to have official support for a read-only git mirror of the > > various FreeBSD projects (mainly src and ports). > > >=20 > No worries, I feel your pain. My Friday evening was ruined too :( >=20 > If you have the resources, *please* try a conversion run of all three > repos too and report your findings. We need to make sure that the > conversion is now more robust and we need to detect drift earlier :/ I wonder if there's a way for the script to determine whether history has been rewritten, causing a force push. If so, it could bail out early, sending an email to interested parties. The email would just say something to the effect of "uh oh, someone needs to pay extra special attention to me!" Then someone could inspect what's going on, determine if a force push is absolutely necessary, and if so, alert the community prior to the force push. Thanks, --=20 Shawn Webb Cofounder and Security Engineer HardenedBSD GPG Key ID: 0x6A84658F52456EEE GPG Key Fingerprint: 2ABA B6BD EF6A F486 BE89 3D9E 6A84 658F 5245 6EEE --3O1VwFp74L81IIeR Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAEBCAAGBQJYQxGyAAoJEGqEZY9SRW7uDK0QAJrCZa9ZHoRNpZub1XhsKeFM YJfcGLazU7HPAIiUKqIwGfMataBOm7TpeEpUbXrJRORH2G2tZpEQC76kZ/UooF98 lqN5W/i4+Fsx+jtbRK0FsqNoNvimoOoqTzj9KgP1Ll+BElM7+zupqLjfFBEVoIAJ 1++IpTUeHxkrIBna3c8/MLl+dBAqBpRT4yFo0ZOVBjnfhgpt+YTIknD7T01BE7+m NPQeIXogcNiBX9qeZ8BGJFaHN3MI59yyiOOMaCWYBslEDCgVuyVIlc5JaUYl+gby Gfq/qBbEFcELSEa9qPmYSinXGD1cJuyx4b2UqufMZ/yjti6ZRBe4Y0izy2cAx5Sz 7vvmPbNonfBSYIffC9+YkLHdUwmU+UnwR63BMq+/kw+3+Xj7yMf7IOdWM2tpeXFa H9oh4Cd2gdXzFAVTHU6Sue6//nybUytrn4K3cruWeG76d8a4b7/QqSC1LcGqUJPa jOOXK6kEKekTxTHdShf4LIXE1BGZxmnDYIzU2zcZc4UkehKfeHFgKHb4M2gVttjd kDxT1eMiLZBx/mWkXK+z12viHuDdIOIZ+3S4j2uavdSlM27MfV1tn7j6YRRrC4ef rwSuiqfAsJmhvBfKbNtRijMEF6iVwtLYlBlZRIEqgFkTAFRXi8VjyBLft7rPdhoB x1DveGxzFdhrbLiWvVWl =zsC2 -----END PGP SIGNATURE----- --3O1VwFp74L81IIeR--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20161203184053.GE82677>