Date: Wed, 5 Sep 2012 20:50:49 -0400 From: Jason Harris <jharris@widomaker.com> To: "Bjoern A. Zeeb" <bzeeb-lists@lists.zabbadoz.net> Cc: Cluster Administrators <clusteradm@freebsd.org>, hubs@freebsd.org Subject: Re: pkgng mirrors Message-ID: <20120906005049.GA20242@laptop> In-Reply-To: <alpine.BSF.2.00.1209051910550.98832@ai.fobar.qr> References: <20120905112013.2d44783c@laptop> <09CB99A0-75BC-426C-BD44-9ACC7CD741D1@exonetric.com> <alpine.BSF.2.00.1209050925180.98832@ai.fobar.qr> <20551.35538.119912.329917@hergotha.csail.mit.edu> <alpine.BSF.2.00.1209051731340.98832@ai.fobar.qr> <20120905190901.GK66364@carrick-users.bishnet.net> <alpine.BSF.2.00.1209051910550.98832@ai.fobar.qr>
next in thread | previous in thread | raw e-mail | index | archive | help
--BXVAT5kNtrzKuDFl Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Wed, Sep 05, 2012 at 07:23:25PM +0000, Bjoern A. Zeeb wrote: > On Wed, 5 Sep 2012, Tim Bishop wrote: =20 > > projects that have their stuff mirrored by third parties. The FreeBSD > > system seems fairly primitive compared to others; whilst you have > > various things to check the status of the mirrors there's nothing (as > > far as I know) automatically happening with it. Other projects have > > systems to automatically remove mirrors from listings if they're not > > updating properly or are missing content. > > > > If FreeBSD was to do something similar you'd be able to retain a well > > established mirror network whilst effectively weeding out broken sites. Sounds like MirrorBrain: http://www.mirrorbrain.org/ I can't believe it hasn't been discussed more in public - Google for "mirrorbrain" site:lists.freebsd.org and find one hit in 2008, 2010, and 2011... :( > Part of the historic problem here is that we have little control over > *.cc.FreeBSD.org currently though we gain more as time passes. So > it's not that easy. We can remove them from website and release lists > and I am not sure to which extend this happening but well worth > investigating. I seem to remember that Release Engineering in the past > had at least made sure that the mirrors mentioned in the release > announcement had the bits at that time. AIUI, MirrorBrain scans the mirrors and does most of this automagically. I really like that it can generate .torrents on the fly with web seeding; that seems perfect for .iso images around release time. > The other problems would however equally remain - no statistics, MB would run on central, project-run webservers and direct users to mirrors with the desired files/collections, so you do know who is requesting what and where they're being sent. As for the server logs, why not remote syslog? Also, I have changed my Apache logs to CSV format and log the server's IP, making it easy to feed logs from multiple servers to a single DB. > maintainance, unable to "atomically" deploy things, and a couple of > other stuff ... The fact that the world has moved by a decade but we > did not may make these changes look more radical than they are. Clearly CDNs do this stuff quite well, but what other Open Source projects have had to resort to using CDNs? > > That said, 10TB worth of packages? That's a lot and I can imagine most > > mirrors couldn't handle that quantity of data. >=20 > 7TB to start within the first year, yes, we are upping the game a > little to give a lot better support to users we hope. It won't be there > from day 1 but that's the direction thigns are heading. A lot of the > mirrors can't cope with the 1G or 2G of ftp, and most of the old stuff > is not on that anymore but on ftp-archive, so yeah... MB could redirect the archived stuff straight to ftp-archive, of course. --=20 Jason Harris | PGP: This _is_ PGP-signed, isn't it? jharris@widomaker.com _|_ Got photons? (TM), (C) 2004 --BXVAT5kNtrzKuDFl Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (FreeBSD) iJ0EARECAF0FAlBH82lWGGh0dHA6Ly9rZXlzZXJ2ZXIua2pzbC5jb206MTEzNzEv cGtzL2xvb2t1cD9vcD1nZXQmc2VhcmNoPTB4RDM5REEwRTMmd2VoYXZleW91bm93 PXRydWUACgkQSypIl9OdoOMt5gCg2SSWhxS+ORWCLReOqkZ/f9DW55kAn1arnRn7 RHZHtZpu4/QCJvXs2GvB =+YfH -----END PGP SIGNATURE----- --BXVAT5kNtrzKuDFl--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20120906005049.GA20242>