% rsync -vaHz --delete rsync://bit0.us-west.freebsd.org/FreeBSD-www-data/ /usr/local/www/
http://rsync.samba.org/, about the various options to be used with rsync. If you sync the whole module (unlike subdirectories), be aware that the module-directory (here "FreeBSD") will not be created, so you cannot omit the target directory. Also you might want to set up a script framework that calls such a command via cron(8).
The FreeBSD website should only be mirrored via rsync.
A command line to mirror the FreeBSD web site might look like:
% rsync -vaHz --delete rsync://bit0.us-west.freebsd.org/FreeBSD-www-data/ /usr/local/www/
Due to very high requirements of bandwidth, storage and adminstration the FreeBSD Project has decided not to allow public mirrors of packages. For sites with lots of machines, it might be advantagous to run a caching HTTP proxy for the pkg(8) process. Alternatively specific packages and their dependencies can be fetched by running something like the following:
% pkg fetch -d -o /usr/local/mirror vim
Once those packages have been fetched, the repository metadata must be generated by running:
% pkg repo /usr/local/mirror
Once the packages have been fetched and the metadata for the repository has been generated, serve the packages up to the client machines via HTTP. For additional information see the man pages for pkg(8), specifically the pkg-repo(8) page.
Every mirror should be updated at a minimum of once per day. Certainly a script with locking to prevent multiple runs happening at the same time will be needed to run from cron(8). Since nearly every admin does this in their own way, specific instructions cannot be provided. It could work something like this:
Put the command to run your mirroring application in a script. Use of a plain /bin/sh
script is recommended.
Add some output redirections so diagnostic messages are logged to a file.
Test if your script works. Check the logs.
Use crontab(1) to add the script to the appropriate user’s crontab(5). This should be a different user than what your FTP daemon runs as so that if file permissions inside your FTP area are not world-readable those files cannot be accessed by anonymous FTP. This is used to "stage" releases - making sure all of the official mirror sites have all of the necessary release files on release day.
Here are some recommended schedules:
FTP fileset: daily
WWW pages: daily
This is an important issue. So this section will spend some effort to explain the backgrounds. We will say this several times: under no circumstances should you mirror from ftp.FreeBSD.org
.
Mirrors are organized by country. All official mirrors have a DNS entry of the form ftpN.CC.FreeBSD.org
. CC (i.e., country code) is the top level domain (TLD) of the country where this mirror is located. N is a number, telling that the host would be the Nth mirror in that country. (Same applies to wwwN.CC.FreeBSD.org
, etc.) There are mirrors with no CC part. These are the mirror sites that are very well connected and allow a large number of concurrent users. ftp.FreeBSD.org
is actually two machines, one currently located in Denmark and the other in the United States. It is NOT a master site and should never be used to mirror from. Lots of online documentation leads "interactive"users to ftp.FreeBSD.org
so automated mirroring systems should find a different machine to mirror from.
Additionally there exists a hierarchy of mirrors, which is described in terms of tiers. The master sites are not referred to but can be described as Tier-0. Mirrors that mirror from these sites can be considered Tier-1, mirrors of Tier-1-mirrors, are Tier-2, etc. Official sites are encouraged to be of a low tier, but the lower the tier the higher the requirements in terms as described in Requirements for FreeBSD Mirrors. Also access to low-tier-mirrors may be restricted, and access to master sites is definitely restricted. The tier-hierarchy is not reflected by DNS and generally not documented anywhere except for the master sites. However, official mirrors with low numbers like 1-4, are usually Tier-1 (this is just a rough hint, and there is no rule).
Under no circumstances should you mirror from ftp.FreeBSD.org
. The short answer is: from the site that is closest to you in Internet terms, or gives you the fastest access.
If you have no special intentions or requirements, the statement in Ok, but Where Should I get the Stuff Now? applies. This means:
Check for those which provide fastest access (number of hops, round-trip-times) and offer the services you intend to use (like rsync).
Contact the administrators of your chosen site stating your request, and asking about their terms and policies.
Set up your mirror as described above.
In general the description in I Just Want to Mirror from Somewhere! still applies. Of course you may want to put some weight on the fact that your upstream should be of a low tier. There are some other considerations about official mirrors that are described in Official Mirrors.
If you have good reasons and good prerequisites, you may want and get access to one of the master sites. Access to these sites is generally restricted, and there are special policies for access. If you are already an official mirror, this certainly helps you getting access. In any other case make sure your country really needs another mirror. If it already has three or more, ask the "zone administrator" (hostmaster@CC.FreeBSD.org) or FreeBSD mirror sites mailing lists first.
Whoever helped you become, an official should have helped you gain access to an appropriate upstream host, either one of the master sites or a suitable Tier-1 site. If not, you can send email to mirror-admin@FreeBSD.org to request help with that.
There is one master site for the FTP fileset.
This is the master site for the FTP fileset.
ftp-master.FreeBSD.org
provides rsync access, in addition to FTP. Refer to Mirroring the FTP Site.
Mirrors are also encouraged to allow rsync access for the FTP contents, since they are Tier-1-mirrors.
Official mirrors are mirrors that
a) have a FreeBSD.org
DNS entry (usually a CNAME).
b) are listed as an official mirror in the FreeBSD documentation (like handbook).
So far to distinguish official mirrors. Official mirrors are not necessarily Tier-1-mirrors. However you probably will not find a Tier-1-mirror, that is not also official.
It is not so easy to state requirements for all official mirrors, since the project is sort of tolerant here. It is more easy to say, what official tier-1 mirrors are required to. All other official mirrors can consider this a big should.
Tier-1 mirrors are required to:
carry the complete fileset
allow access to other mirror sites
provide FTP and rsync access
Furthermore, admins should be subscribed to the FreeBSD mirror sites mailing lists. See this link for details, how to subscribe.
It is very important for a hub administrator, especially Tier-1 hub admins, to check the release schedule for the next FreeBSD release. This is important because it will tell you when the next release is scheduled to come out, and thus giving you time to prepare for the big spike of traffic which follows it. It is also important that hub administrators try to keep their mirrors as up-to-date as possible (again, even more crucial for Tier-1 mirrors). If Mirror1 does not update for a while, lower tier mirrors will begin to mirror old data from Mirror1 and thus begins a downward spiral… Keep your mirrors up to date! |
We are not accepting any new mirrors at this time.
Here are links to the stat pages of your favorite mirrors (aka the only ones who feel like providing stats).
ftp.is.FreeBSD.org - hostmaster@is.FreeBSD.org - (Bandwidth) (FTP processes) (HTTP processes)
ftp2.ru.FreeBSD.org - mirror@macomnet.ru - (Bandwidth) (HTTP and FTP users)