Date: Mon, 24 Sep 2018 09:34:55 +0000 (UTC) From: Benedict Reuschling <bcr@FreeBSD.org> To: doc-committers@freebsd.org, svn-doc-all@freebsd.org, svn-doc-head@freebsd.org Subject: svn commit: r52296 - head/en_US.ISO8859-1/articles/hubs Message-ID: <201809240934.w8O9YtVH074452@repo.freebsd.org>
next in thread | raw e-mail | index | archive | help
Author: bcr Date: Mon Sep 24 09:34:54 2018 New Revision: 52296 URL: https://svnweb.freebsd.org/changeset/doc/52296 Log: Clean up the article from textproc/igor warnings such as: - wrap long lines - use tabs instead of spaces - capitalization - put content after <para> tags - leave a blank line after <title> tags - use two spaces at sentence start Modified: head/en_US.ISO8859-1/articles/hubs/article.xml Modified: head/en_US.ISO8859-1/articles/hubs/article.xml ============================================================================== --- head/en_US.ISO8859-1/articles/hubs/article.xml Sun Sep 23 18:41:53 2018 (r52295) +++ head/en_US.ISO8859-1/articles/hubs/article.xml Mon Sep 24 09:34:54 2018 (r52296) @@ -1,22 +1,57 @@ <?xml version="1.0" encoding="iso-8859-1"?> <!DOCTYPE article PUBLIC "-//FreeBSD//DTD DocBook XML V5.0-Based Extension//EN" "http://www.FreeBSD.org/XML/share/xml/freebsd50.dtd">; -<article xmlns="http://docbook.org/ns/docbook" xmlns:xlink="http://www.w3.org/1999/xlink" version="5.0" xml:lang="en"> - <info><title>Mirroring FreeBSD</title> - +<article xmlns="http://docbook.org/ns/docbook" + xmlns:xlink="http://www.w3.org/1999/xlink" version="5.0" + xml:lang="en"> + <info> + <title>Mirroring FreeBSD</title> + <authorgroup> - <author><personname><firstname>Jun</firstname><surname>Kuriyama</surname></personname><affiliation> - <address><email>kuriyama@FreeBSD.org</email></address> - </affiliation></author> - <author><personname><firstname>Valentino</firstname><surname>Vaschetto</surname></personname><affiliation> - <address><email>logo@FreeBSD.org</email></address> - </affiliation></author> - <author><personname><firstname>Daniel</firstname><surname>Lang</surname></personname><affiliation> - <address><email>dl@leo.org</email></address> - </affiliation></author> - <author><personname><firstname>Ken</firstname><surname>Smith</surname></personname><affiliation> - <address><email>kensmith@FreeBSD.org</email></address> - </affiliation></author> + <author> + <personname> + <firstname>Jun</firstname> + <surname>Kuriyama</surname> + </personname> + <affiliation> + <address> + <email>kuriyama@FreeBSD.org</email> + </address> + </affiliation> + </author> + <author> + <personname> + <firstname>Valentino</firstname> + <surname>Vaschetto</surname> + </personname> + <affiliation> + <address> + <email>logo@FreeBSD.org</email> + </address> + </affiliation> + </author> + <author> + <personname> + <firstname>Daniel</firstname> + <surname>Lang</surname> + </personname> + <affiliation> + <address> + <email>dl@leo.org</email> + </address> + </affiliation> + </author> + <author> + <personname> + <firstname>Ken</firstname> + <surname>Smith</surname> + </personname> + <affiliation> + <address> + <email>kensmith@FreeBSD.org</email> + </address> + </affiliation> + </author> </authorgroup> <legalnotice xml:id="trademarks" role="trademarks"> @@ -30,7 +65,7 @@ <abstract> <para>An in-progress article on how to mirror FreeBSD, aimed at - hub administrators.</para> + hub administrators.</para> </abstract> </info> @@ -47,662 +82,723 @@ </sect1> <sect1 xml:id="mirror-requirements"> - <title>Requirements for FreeBSD mirrors</title> + <title>Requirements for FreeBSD Mirrors</title> + <sect2 xml:id="mirror-diskspace"> <title>Disk Space</title> - <para> - Disk space is one of the most important requirements. - Depending on the set of releases, architectures, - and degree of completeness you want to mirror, a huge - amount of disk space may be consumed. Also keep in mind - that <emphasis>official</emphasis> mirrors are probably required to be - complete. The web pages should - always be mirrored completely. Also note that the - numbers stated here are reflecting the current - state (at &rel2.current;-RELEASE/&rel.current;-RELEASE). Further development and - releases will only increase the required amount. - Also make sure to keep some (ca. 10-20%) extra space - around just to be sure. - Here are some approximate figures: - </para> + + <para>Disk space is one of the most important requirements. + Depending on the set of releases, architectures, and degree of + completeness you want to mirror, a huge amount of disk space + may be consumed. Also keep in mind that + <emphasis>official</emphasis> mirrors are probably required to + be complete. The web pages should always be mirrored + completely. Also note that the numbers stated here are + reflecting the current state (at + &rel2.current;-RELEASE/&rel.current;-RELEASE). Further + development and releases will only increase the required + amount. Also make sure to keep some (ca. 10-20%) extra space + around just to be sure. Here are some approximate + figures:</para> + <itemizedlist> - <listitem><para>Full FTP Distribution: 1.4 TB</para></listitem> - <listitem><para>CTM deltas: 10 GB</para></listitem> - <listitem><para>Web pages: 1GB</para></listitem> + <listitem> + <para>Full FTP Distribution: 1.4 TB</para> + </listitem> + <listitem> + <para>CTM deltas: 10 GB</para> + </listitem> + <listitem> + <para>Web pages: 1GB</para> + </listitem> </itemizedlist> - <para> - The current disk usage of FTP Distribution can be found at - <link xlink:href="ftp://ftp.FreeBSD.org/pub/FreeBSD/dir.sizes">ftp://ftp.FreeBSD.org/pub/FreeBSD/dir.sizes</link>. - </para> + + <para>The current disk usage of FTP Distribution can be found at + <link + xlink:href="ftp://ftp.FreeBSD.org/pub/FreeBSD/dir.sizes">ftp://ftp.FreeBSD.org/pub/FreeBSD/dir.sizes</link>.</para>; </sect2> <sect2 xml:id="mirror-bandwidth"> <title>Network Connection/Bandwidth</title> - <para> - Of course, you need to be connected to the Internet. - The required bandwidth depends on your intended use - of the mirror. If you just want to mirror some - parts of FreeBSD for local use at your site/intranet, - the demand may be much smaller than if you want to - make the files publicly available. If you intend - to become an official mirror, the bandwidth required will be even higher. We can only give rough - estimates here: - </para> + + <para>Of course, you need to be connected to the Internet. The + required bandwidth depends on your intended use of the mirror. + If you just want to mirror some parts of FreeBSD for local use + at your site/intranet, the demand may be much smaller than if + you want to make the files publicly available. If you intend + to become an official mirror, the bandwidth required will be + even higher. We can only give rough estimates here:</para> + <itemizedlist> - <listitem><para>Local site, no public access: basically no minimum, - but < 2 Mbps could make syncing too slow.</para></listitem> - <listitem><para>Unofficial public site: 34 Mbps is probably a good start.</para></listitem> - <listitem><para>Official site: > 100 Mbps is recommended, and your host - should be connected as close as possible to your border router.</para></listitem> + <listitem> + <para>Local site, no public access: basically no minimum, + but < 2 Mbps could make syncing too + slow.</para> + </listitem> + + <listitem> + <para>Unofficial public site: 34 Mbps is probably a good + start.</para> + </listitem> + + <listitem> + <para>Official site: > 100 Mbps is recommended, and your + host should be connected as close as possible to your + border router.</para> + </listitem> </itemizedlist> </sect2> + <sect2 xml:id="mirror-system"> <title>System Requirements, CPU, RAM</title> - <para> - One thing this depends on the expected number of clients, - which is determined by the server's policy. It is - also affected by the types of services you want to offer. - Plain FTP or HTTP services may not require a huge - amount of resources. Watch out if you provide - rsync. This can have a huge - impact on CPU and memory requirements as it is - considered a memory hog. - The following - are just examples to give you a very rough hint. - </para> - <para> - For a moderately visited site that offers - <application>rsync</application>, you might - consider a current CPU with around 800MHz - 1 GHz, - and at least 512MB RAM. This is probably the - minimum you want for an <emphasis>official</emphasis> - site. - </para> - <para> - For a frequently used site you definitely need - more RAM (consider 2GB as a good start) - and possibly more CPU, which could also mean - that you need to go for a SMP system. - </para> - <para> - You also want to consider a fast disk subsystem. - Operations on the SVN repository require a fast - disk subsystem (RAID is highly advised). A SCSI - controller that has a cache of its own can also - speed up things since most of these services incur a - large number of small modifications to the disk. - </para> + + <para>One thing this depends on the expected number of clients, + which is determined by the server's policy. It is also + affected by the types of services you want to offer. Plain + FTP or HTTP services may not require a huge amount of + resources. Watch out if you provide rsync. This can have a + huge impact on CPU and memory requirements as it is considered + a memory hog. The following are just examples to give you a + very rough hint.</para> + + <para>For a moderately visited site that offers + <application>rsync</application>, you might consider a current + CPU with around 800MHz - 1 GHz, and at least 512MB RAM. This + is probably the minimum you want for an + <emphasis>official</emphasis> site.</para> + + <para>For a frequently used site you definitely need more RAM + (consider 2GB as a good start) and possibly more CPU, which + could also mean that you need to go for a SMP system.</para> + + <para>You also want to consider a fast disk subsystem. + Operations on the SVN repository require a fast disk subsystem + (RAID is highly advised). A SCSI controller that has a cache + of its own can also speed up things since most of these + services incur a large number of small modifications to the + disk.</para> </sect2> + <sect2 xml:id="mirror-services"> - <title>Services to offer</title> - <para> - Every mirror site is required to have a set of core services - available. In addition to these required services, there are - a number of optional services that - server administrators may choose to offer. This section explains - which services you can provide and how to go about implementing them. - </para> + <title>Services to Offer</title> + + <para>Every mirror site is required to have a set of core + services available. In addition to these required services, + there are a number of optional services that server + administrators may choose to offer. This section explains + which services you can provide and how to go about + implementing them.</para> + <sect3 xml:id="mirror-serv-ftp"> - <title>FTP (required for FTP fileset)</title> - <para> - This is one of the most basic services, and - it is required for each mirror offering public - FTP distributions. FTP access must be - anonymous, and no upload/download ratios - are allowed (a ridiculous thing anyway). - Upload capability is not required (and <emphasis>must</emphasis> - never be allowed for the FreeBSD file space). - Also the FreeBSD archive should be available under - the path <filename>/pub/FreeBSD</filename>. - </para> - <para> - There is a lot of software available which - can be set up to allow anonymous FTP - (in alphabetical order).</para> - <itemizedlist> - <listitem><para><command>/usr/libexec/ftpd</command>: FreeBSD's own ftpd - can be used. Be sure to read &man.ftpd.8;.</para> - </listitem> - <listitem> - <para><package>ftp/ncftpd</package>: A commercial package, - free for educational use.</para> - </listitem> - <listitem> - <para><package>ftp/oftpd</package>: An ftpd designed with - security as a main focus.</para> - </listitem> - <listitem> - <para><package>ftp/proftpd</package>: A modular and very flexible ftpd.</para> - </listitem> - <listitem> - <para><package>ftp/pure-ftpd</package>: Another ftpd developed with - security in mind.</para> - </listitem> - <listitem><para><package>ftp/twoftpd</package>: As above.</para></listitem> - <listitem><para><package>ftp/vsftpd</package>: The <quote>very secure</quote> ftpd.</para></listitem> - </itemizedlist> - <para>FreeBSD's <application>ftpd</application>, <application>proftpd</application> - and maybe <application>ncftpd</application> - are among the most commonly used FTPds. - The others do not have a large userbase among mirror sites. One - thing to consider is that you may need flexibility in limiting - how many simultaneous connections are allowed, thus limiting how - much network bandwidth and system resources are consumed. - </para> + <title>FTP (required for FTP Fileset)</title> + + <para>This is one of the most basic services, and it is + required for each mirror offering public FTP distributions. + FTP access must be anonymous, and no upload/download ratios + are allowed (a ridiculous thing anyway). Upload capability + is not required (and <emphasis>must</emphasis> never be + allowed for the FreeBSD file space). Also the FreeBSD + archive should be available under the path + <filename>/pub/FreeBSD</filename>.</para> + + <para>There is a lot of software available which can be set up + to allow anonymous FTP (in alphabetical order).</para> + + <itemizedlist> + <listitem> + <para><command>/usr/libexec/ftpd</command>: FreeBSD's own + ftpd can be used. Be sure to read &man.ftpd.8;.</para> + </listitem> + + <listitem> + <para><package>ftp/ncftpd</package>: A commercial package, + free for educational use.</para> + </listitem> + + <listitem> + <para><package>ftp/oftpd</package>: An ftpd designed with + security as a main focus.</para> + </listitem> + + <listitem> + <para><package>ftp/proftpd</package>: A modular and very + flexible ftpd.</para> + </listitem> + + <listitem> + <para><package>ftp/pure-ftpd</package>: Another ftpd + developed with security in mind.</para> + </listitem> + + <listitem> + <para><package>ftp/twoftpd</package>: As + above.</para> + </listitem> + + <listitem> + <para><package>ftp/vsftpd</package>: The <quote>very + secure</quote> ftpd.</para> + </listitem> + </itemizedlist> + + <para>FreeBSD's <application>ftpd</application>, + <application>proftpd</application> and maybe + <application>ncftpd</application> are among the most + commonly used FTPds. The others do not have a large + userbase among mirror sites. One thing to consider is that + you may need flexibility in limiting how many simultaneous + connections are allowed, thus limiting how much network + bandwidth and system resources are consumed.</para> </sect3> + <sect3 xml:id="mirror-serv-rsync"> - <title>Rsync (optional for FTP fileset)</title> - <para> - <application>Rsync</application> is often offered for access to the - contents of the FTP area of FreeBSD, so other mirror sites can use your system as their source. The - protocol is different from FTP in many ways. - It is much more - bandwidth friendly, as only differences between files - are transferred instead of whole files when they change. - <application>Rsync</application> does require a significant amount of memory for - each instance. The size depends on the size of - the synced module in terms of the number of directories and - files. <application>Rsync</application> can use <command>rsh</command> and - <command>ssh</command> (now default) as a transport, - or use its own protocol for stand-alone access - (this is the preferred method for public rsync servers). - Authentication, connection limits, and other restrictions - may be applied. There is just one software package - available:</para> - <itemizedlist> - <listitem><para><package>net/rsync</package></para></listitem> - </itemizedlist> + <title>Rsync (optional for FTP Fileset)</title> + + <para><application>Rsync</application> is often offered for + access to the contents of the FTP area of FreeBSD, so other + mirror sites can use your system as their source. The + protocol is different from FTP in many ways. It is much + more bandwidth friendly, as only differences between files + are transferred instead of whole files when they change. + <application>Rsync</application> does require a significant + amount of memory for each instance. The size depends on the + size of the synced module in terms of the number of + directories and files. <application>Rsync</application> can + use <command>rsh</command> and <command>ssh</command> (now + default) as a transport, or use its own protocol for + stand-alone access (this is the preferred method for public + rsync servers). Authentication, connection limits, and + other restrictions may be applied. There is just one + software package available:</para> + + <itemizedlist> + <listitem> + <para><package>net/rsync</package></para> + </listitem> + </itemizedlist> </sect3> + <sect3 xml:id="mirror-serv-http"> - <title>HTTP (required for web pages, optional for FTP fileset)</title> - <para> - If you want to offer the FreeBSD web pages, you will need - to install a web server. - You may optionally offer the FTP fileset via HTTP. - The choice of web server software is left up to the mirror administrator. - Some of the most popular choices are:</para> + <title>HTTP (required for Web Pages, Optional for FTP + Fileset)</title> - <itemizedlist> - <listitem> - <para><package>www/apache24</package>: - <application>Apache</application> is still one of the most widely - deployed web servers on the Internet. It is used - extensively by the FreeBSD Project.</para> - </listitem> + <para>If you want to offer the FreeBSD web pages, you will + need to install a web server. You may optionally offer the + FTP fileset via HTTP. The choice of web server software is + left up to the mirror administrator. Some of the most + popular choices are:</para> - <listitem> - <para><package>www/boa</package>: - <application>Boa</application> is a single-tasking HTTP server. - Unlike traditional web servers, it does not fork for each incoming - connection, nor does it fork many copies of itself to handle multiple - connections. Although, it should provide considerably great - performance for purely static content.</para> - </listitem> + <itemizedlist> + <listitem> + <para><package>www/apache24</package>: + <application>Apache</application> is still one of the + most widely deployed web servers on the Internet. It is + used extensively by the FreeBSD Project.</para> + </listitem> - <listitem> - <para><package>www/cherokee</package>: - <application>>Cherokee</application> is a very fast, flexible and - easy to configure web server. It supports the widespread technologies - nowadays: FastCGI, SCGI, PHP, CGI, SSL/TLS encrypted connections, - vhosts, users authentication, on the fly encoding and load balancing. - It also generates <application>Apache</application> compatible log - files.</para> - </listitem> + <listitem> + <para><package>www/boa</package>: + <application>Boa</application> is a single-tasking HTTP + server. Unlike traditional web servers, it does not + fork for each incoming connection, nor does it fork many + copies of itself to handle multiple connections. + Although, it should provide considerably great + performance for purely static content.</para> + </listitem> - <listitem> - <para><package>www/lighttpd</package>: - <application>lighttpd</application> is a secure, fast, compliant and - very flexible web server which has been optimized for high-performance - environments. It has a very low memory footprint compared to other web - servers and takes care of cpu-load.</para> - </listitem> + <listitem> + <para><package>www/cherokee</package>: + <application>>Cherokee</application> is a very fast, + flexible and easy to configure web server. It supports + the widespread technologies nowadays: FastCGI, SCGI, + PHP, CGI, SSL/TLS encrypted connections, vhosts, users + authentication, on the fly encoding and load balancing. + It also generates <application>Apache</application> + compatible log files.</para> + </listitem> - <listitem> - <para><package>www/nginx</package>: - <application>nginx</application> is a high performance edge web - server with a low memory footprint and key features to build - a modern and efficient web infrastructure. Features include - a HTTP server, HTTP and mail reverse proxy, caching, load - balancing, compression, request throttling, connection - multiplexing and reuse, SSL offload and HTTP media - streaming.</para> - </listitem> + <listitem> + <para><package>www/lighttpd</package>: + <application>lighttpd</application> is a secure, fast, + compliant and very flexible web server which has been + optimized for high-performance environments. It has a + very low memory footprint compared to other web servers + and takes care of cpu-load.</para> + </listitem> - <listitem> - <para><package>www/thttpd</package>: - If you are going to be serving a large amount of static content - you may find that using an application such as - <application>thttpd</application> is more efficient than others. - It is also optimized for excellent performance on FreeBSD.</para> - </listitem> - </itemizedlist> + <listitem> + <para><package>www/nginx</package>: + <application>nginx</application> is a high performance + edge web server with a low memory footprint and key + features to build a modern and efficient web + infrastructure. Features include a HTTP server, HTTP + and mail reverse proxy, caching, load balancing, + compression, request throttling, connection multiplexing + and reuse, SSL offload and HTTP media streaming.</para> + </listitem> + + <listitem> + <para><package>www/thttpd</package>: If you are going to + be serving a large amount of static content you may find + that using an application such as + <application>thttpd</application> is more efficient than + others. It is also optimized for excellent performance + on FreeBSD.</para> + </listitem> + </itemizedlist> </sect3> - </sect2> + </sect2> </sect1> + <sect1 xml:id="mirror-howto"> <title>How to Mirror FreeBSD</title> - <para> - Ok, now you know the requirements and how to offer - the services, but not how to get it. :-) - This section explains how to actually mirror - the various parts of FreeBSD, what tools to use, - and where to mirror from. - </para> + + <para>Ok, now you know the requirements and how to offer the + services, but not how to get it. :-) This section explains how + to actually mirror the various parts of FreeBSD, what tools to + use, and where to mirror from.</para> + <sect2 xml:id="mirror-ftp-rsync"> - <title>Mirroring the FTP site</title> - <para> - The FTP area is the largest amount of data that - needs to be mirrored. It includes the <emphasis>distribution - sets</emphasis> required for network installation, the - <emphasis>branches</emphasis> which are actually snapshots - of checked-out source trees, the <emphasis>ISO Images</emphasis> - to write CD-ROMs with the installation distribution, - a live file system, and a snapshot of the ports tree. All of - course for various FreeBSD versions, and various architectures. - </para> - <para> - The best way to mirror the FTP area is <application>rsync</application>. - You can install the port <package>net/rsync</package> and then use - rsync to sync with your upstream host. - <application>rsync</application> is already mentioned - in <xref linkend="mirror-serv-rsync"/>. - Since <application>rsync</application> access is not - required, your preferred upstream site may not allow it. - You may need to hunt around a little bit to find a site - that allows <application>rsync</application> access.</para> - <note> - <para> - Since the number of <application>rsync</application> - clients will have a significant impact on the server - machine, most admins impose limitations on their - server. For a mirror, you should ask the site maintainer - you are syncing from about their policy, and maybe - an exception for your host (since you are a mirror). - </para> - </note> - <para>A command line to mirror FreeBSD might look like:</para> - <screen>&prompt.user; <userinput>rsync -vaHz --delete rsync://ftp4.de.FreeBSD.org/FreeBSD/ /pub/FreeBSD/</userinput></screen> - <para>Consult the documentation for <application>rsync</application>, - which is also available at - <link xlink:href="http://rsync.samba.org/">http://rsync.samba.org/</link>, - about the various options to be used with rsync. - If you sync the whole module (unlike subdirectories), - be aware that the module-directory (here "FreeBSD") - will not be created, so you cannot omit the target directory. - Also you might - want to set up a script framework that calls such a command - via &man.cron.8;. - </para> + <title>Mirroring the FTP Site</title> + + <para>The FTP area is the largest amount of data that needs to + be mirrored. It includes the <emphasis>distribution + sets</emphasis> required for network installation, the + <emphasis>branches</emphasis> which are actually snapshots of + checked-out source trees, the <emphasis>ISO Images</emphasis> + to write CD-ROMs with the installation distribution, a live + file system, and a snapshot of the ports tree. All of course + for various FreeBSD versions, and various + architectures.</para> + + <para>The best way to mirror the FTP area is + <application>rsync</application>. You can install the port + <package>net/rsync</package> and then use rsync to sync with + your upstream host. <application>rsync</application> is + already mentioned in <xref linkend="mirror-serv-rsync"/>. + Since <application>rsync</application> access is not required, + your preferred upstream site may not allow it. You may need + to hunt around a little bit to find a site that allows + <application>rsync</application> access.</para> + + <note> + <para>Since the number of <application>rsync</application> + clients will have a significant impact on the server + machine, most admins impose limitations on their server. + For a mirror, you should ask the site maintainer you are + syncing from about their policy, and maybe an exception for + your host (since you are a mirror).</para> + </note> + + <para>A command line to mirror FreeBSD might look like:</para> + + <screen>&prompt.user; <userinput>rsync -vaHz --delete rsync://ftp4.de.FreeBSD.org/FreeBSD/ /pub/FreeBSD/</userinput></screen> + + <para>Consult the documentation for + <application>rsync</application>, which is also available at + <link + xlink:href="http://rsync.samba.org/">http://rsync.samba.org/</link>, + about the various options to be used with rsync. If you sync + the whole module (unlike subdirectories), be aware that the + module-directory (here "FreeBSD") will not be created, so you + cannot omit the target directory. Also you might want to set + up a script framework that calls such a command via + &man.cron.8;.</para> </sect2> + <sect2 xml:id="mirror-www"> - <title>Mirroring the WWW pages</title> - <para> - The FreeBSD website should only be mirrored via + <title>Mirroring the WWW Pages</title> + + <para>The FreeBSD website should only be mirrored via <application>rsync</application>.</para> - <para>A command line to mirror the FreeBSD web site might look like:</para> + + <para>A command line to mirror the FreeBSD web site might look + like:</para> + <screen>&prompt.user; <userinput>rsync -vaHz --delete rsync://bit0.us-west.freebsd.org/FreeBSD-www-data/ /usr/local/www/</userinput></screen> - </sect2> - <sect2 xml:id="mirror-pkgs"> - <title>Mirroring Packages</title> - <para>Due to very high requirements of bandwidth, storage and - adminstration the &os; Project has decided not to allow public - mirrors of packages. For sites with lots of machines, it might - be advantagous to run a caching HTTP proxy for the &man.pkg.8; - process. Alternatively specific packages and their dependencies - can be fetched by running something like the following:</para> + </sect2> - <screen>&prompt.user; <userinput>pkg fetch -d -o <replaceable>/usr/local/mirror</replaceable> <replaceable>vim</replaceable></userinput></screen> + <sect2 xml:id="mirror-pkgs"> + <title>Mirroring Packages</title> - <para>Once those packages have been fetched, the repository metadata must be generated by running:</para> + <para>Due to very high requirements of bandwidth, storage and + adminstration the &os; Project has decided not to allow public + mirrors of packages. For sites with lots of machines, it + might be advantagous to run a caching HTTP proxy for the + &man.pkg.8; process. Alternatively specific packages and + their dependencies can be fetched by running something like + the following:</para> - <screen>&prompt.user; <userinput>pkg repo <replaceable>/usr/local/mirror</replaceable></userinput></screen> + <screen>&prompt.user; <userinput>pkg fetch -d -o <replaceable>/usr/local/mirror</replaceable> <replaceable>vim</replaceable></userinput></screen> - <para>Once the packages have been fetched and the metadata for the - repository has been generated, serve the packages up to the - client machines via HTTP. For additional information see the - man pages for &man.pkg.8;, specifically the &man.pkg-repo.8; page. - </para> - </sect2> - <sect2 xml:id="mirror-how-often"> - <title>How often should I mirror?</title> - <para> - Every mirror should be updated at a minimum of once per day. - Certainly a script with locking to prevent multiple runs - happening at the same time will be needed to run from - &man.cron.8;. Since nearly every admin does this in their own - way, specific instructions cannot be provided. It could work - something like this: - </para> - <procedure> - <step> - <para> - Put the command to run your mirroring application - in a script. Use of a plain <command>/bin/sh</command> - script is recommended. - </para> - </step> - <step> - <para> - Add some output redirections so diagnostic - messages are logged to a file. - </para> - </step> - <step> - <para> - Test if your script works. Check the logs. - </para> - </step> - <step> - <para> - Use &man.crontab.1; to add the script to the - appropriate user's &man.crontab.5;. This should be a - different user than what your FTP daemon runs as so that - if file permissions inside your FTP area are not - world-readable those files can not be accessed by anonymous - FTP. This is used to <quote>stage</quote> releases — - making sure all of the official mirror sites have all of the - necessary release files on release day. - </para> - </step> - </procedure> - <para> - Here are some recommended schedules:</para> - <itemizedlist> - <listitem><para>FTP fileset: daily</para></listitem> - <listitem><para>WWW pages: daily</para></listitem> - </itemizedlist> - </sect2> + <para>Once those packages have been fetched, the repository + metadata must be generated by running:</para> + + <screen>&prompt.user; <userinput>pkg repo <replaceable>/usr/local/mirror</replaceable></userinput></screen> + + <para>Once the packages have been fetched and the metadata for + the repository has been generated, serve the packages up to + the client machines via HTTP. For additional information see + the man pages for &man.pkg.8;, specifically the + &man.pkg-repo.8; page.</para> + </sect2> + + <sect2 xml:id="mirror-how-often"> + <title>How Often Should I Mirror?</title> + + <para>Every mirror should be updated at a minimum of once per + day. Certainly a script with locking to prevent multiple runs + happening at the same time will be needed to run from + &man.cron.8;. Since nearly every admin does this in their own + way, specific instructions cannot be provided. It could work + something like this:</para> + + <procedure> + <step> + <para>Put the command to run your mirroring application in a + script. Use of a plain <command>/bin/sh</command> script + is recommended.</para> + </step> + + <step> + <para>Add some output redirections so diagnostic messages + are logged to a file.</para> + </step> + + <step> + <para>Test if your script works. Check the logs.</para> + </step> + + <step> + <para>Use &man.crontab.1; to add the script to the + appropriate user's &man.crontab.5;. This should be a + different user than what your FTP daemon runs as so that + if file permissions inside your FTP area are not + world-readable those files cannot be accessed by anonymous + FTP. This is used to <quote>stage</quote> releases + — making sure all of the official mirror sites have + all of the necessary release files on release day.</para> + </step> + </procedure> + + <para>Here are some recommended schedules:</para> + + <itemizedlist> + <listitem> + <para>FTP fileset: daily</para> + </listitem> + <listitem> + <para>WWW pages: daily</para> + </listitem> + </itemizedlist> + </sect2> </sect1> + <sect1 xml:id="mirror-where"> - <title>Where to mirror from</title> - <para> - This is an important issue. So this section will - spend some effort to explain the backgrounds. We will say this - several times: under no circumstances should you mirror from - <systemitem class="fqdomainname">ftp.FreeBSD.org</systemitem>. - </para> + <title>Where to Mirror From</title> + + <para>This is an important issue. So this section will spend some + effort to explain the backgrounds. We will say this several + times: under no circumstances should you mirror from <systemitem + class="fqdomainname">ftp.FreeBSD.org</systemitem>.</para> + <sect2 xml:id="mirror-where-organization"> - <title>A few words about the organization</title> - <para> - Mirrors are organized by country. All - official mirrors have a DNS entry of the form - <systemitem class="fqdomainname">ftpN.CC.FreeBSD.org</systemitem>. - <emphasis>CC</emphasis> (i.e. country code) is the - <emphasis>top level domain</emphasis> (TLD) - of the country where this mirror is located. - <emphasis>N</emphasis> is a number, - telling that the host would be the <emphasis>Nth</emphasis> - mirror in that country. - (Same applies to - <systemitem>wwwN.CC.FreeBSD.org</systemitem>, etc.) - There are mirrors with no <emphasis>CC</emphasis> part. - These are the mirror sites that are very well connected and - allow a large number of concurrent users. - <systemitem class="fqdomainname">ftp.FreeBSD.org</systemitem> is actually two machines, one currently - located in Denmark and the other in the United States. - It is <emphasis>NOT</emphasis> a master site and should never be - used to mirror from. Lots of online documentation leads - <quote>interactive</quote>users to - <systemitem class="fqdomainname">ftp.FreeBSD.org</systemitem> so automated mirroring - systems should find a different machine to mirror from. - </para> - <para> - Additionally there exists a hierarchy of mirrors, which - is described in terms of <emphasis>tiers</emphasis>. - The master sites are not referred to but can be - described as <emphasis>Tier-0</emphasis>. Mirrors - that mirror from these sites can be considered - <emphasis>Tier-1</emphasis>, mirrors of <emphasis>Tier-1</emphasis>-mirrors, - are <emphasis>Tier-2</emphasis>, etc. - Official sites are encouraged to be of a low <emphasis>tier</emphasis>, - but the lower the tier the higher the requirements in - terms as described in <xref linkend="mirror-requirements"/>. - Also access to low-tier-mirrors may be restricted, and - access to master sites is definitely restricted. - The <emphasis>tier</emphasis>-hierarchy is not reflected - by DNS and generally not documented anywhere except - for the master sites. However, official mirrors with low numbers - like 1-4, are usually <emphasis>Tier-1</emphasis> - (this is just a rough hint, and there is no rule). - </para> + <title>A few Words About the Organization</title> + + <para>Mirrors are organized by country. All official mirrors + have a DNS entry of the form <systemitem + class="fqdomainname">ftpN.CC.FreeBSD.org</systemitem>. + <emphasis>CC</emphasis> (i.e., country code) is the + <emphasis>top level domain</emphasis> (TLD) of the country + where this mirror is located. <emphasis>N</emphasis> is a + number, telling that the host would be the + <emphasis>Nth</emphasis> mirror in that country. (Same + applies to <systemitem>wwwN.CC.FreeBSD.org</systemitem>, etc.) + There are mirrors with no <emphasis>CC</emphasis> part. These + are the mirror sites that are very well connected and allow a + large number of concurrent users. <systemitem + class="fqdomainname">ftp.FreeBSD.org</systemitem> is + actually two machines, one currently located in Denmark and + the other in the United States. It is + <emphasis>NOT</emphasis> a master site and should never be + used to mirror from. Lots of online documentation leads + <quote>interactive</quote>users to <systemitem + class="fqdomainname">ftp.FreeBSD.org</systemitem> so + automated mirroring systems should find a different machine to + mirror from.</para> + + <para>Additionally there exists a hierarchy of mirrors, which is + described in terms of <emphasis>tiers</emphasis>. The master + sites are not referred to but can be described as + <emphasis>Tier-0</emphasis>. Mirrors that mirror from these + sites can be considered <emphasis>Tier-1</emphasis>, mirrors + of <emphasis>Tier-1</emphasis>-mirrors, are + <emphasis>Tier-2</emphasis>, etc. Official sites are + encouraged to be of a low <emphasis>tier</emphasis>, but the + lower the tier the higher the requirements in terms as + described in <xref linkend="mirror-requirements"/>. Also + access to low-tier-mirrors may be restricted, and access to + master sites is definitely restricted. The + <emphasis>tier</emphasis>-hierarchy is not reflected by DNS + and generally not documented anywhere except for the master + sites. However, official mirrors with low numbers like 1-4, + are usually <emphasis>Tier-1</emphasis> (this is just a rough + hint, and there is no rule).</para> </sect2> + <sect2 xml:id="mirror-where-where"> - <title>Ok, but where should I get the stuff now?</title> - <para> - Under no circumstances should you mirror from <systemitem class="fqdomainname">ftp.FreeBSD.org</systemitem>. - The short answer is: from the - site that is closest to you in Internet terms, or gives you - the fastest access. - </para> + <title>Ok, but Where Should I get the Stuff Now?</title> + + <para>Under no circumstances should you mirror from <systemitem + class="fqdomainname">ftp.FreeBSD.org</systemitem>. The + short answer is: from the site that is closest to you in + Internet terms, or gives you the fastest access.</para> + <sect3 xml:id="mirror-where-simple"> - <title>I just want to mirror from somewhere!</title> - <para> - If you have no special intentions or - requirements, the statement in <xref linkend="mirror-where-where"/> - applies. This means: - </para> - <procedure> - <step> - <para> - Check for those which provide fastest access - (number of hops, round-trip-times) - and offer the services you intend to - use (like <application>rsync</application>). - </para> - </step> - <step> - <para> - Contact the administrators of your chosen site stating your - request, and asking about their terms and - policies. - </para> - </step> - <step> - <para> - Set up your mirror as described above. - </para> - </step> - </procedure> + <title>I Just Want to Mirror from Somewhere!</title> + + <para>If you have no special intentions or requirements, the + statement in <xref linkend="mirror-where-where"/> applies. + This means:</para> + + <procedure> + <step> + <para>Check for those which provide fastest access (number + of hops, round-trip-times) and offer the services you + intend to use (like + <application>rsync</application>).</para> + </step> + + <step> + <para>Contact the administrators of your chosen site + stating your request, and asking about their terms and + policies.</para> + </step> + + <step> + <para>Set up your mirror as described above.</para> + </step> + </procedure> </sect3> + <sect3 xml:id="mirror-where-official"> - <title>I am an official mirror, what is the right site for me?</title> - <para> *** DIFF OUTPUT TRUNCATED AT 1000 LINES ***
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?201809240934.w8O9YtVH074452>