Date: Fri, 4 Oct 2013 18:55:55 +0000 (UTC) From: Glen Barber <gjb@FreeBSD.org> To: doc-committers@freebsd.org, svn-doc-all@freebsd.org, svn-doc-head@freebsd.org Subject: svn commit: r42830 - head/en_US.ISO8859-1/htdocs Message-ID: <201310041855.r94IttLG082111@svn.freebsd.org>
next in thread | raw e-mail | index | archive | help
Author: gjb Date: Fri Oct 4 18:55:55 2013 New Revision: 42830 URL: http://svnweb.freebsd.org/changeset/doc/42830 Log: Remove a stray blank line from robots.txt. Add a commented Crawl-delay of 10 seconds, which might be useful after the commit that is to follow. Modified: head/en_US.ISO8859-1/htdocs/robots.txt Modified: head/en_US.ISO8859-1/htdocs/robots.txt ============================================================================== --- head/en_US.ISO8859-1/htdocs/robots.txt Fri Oct 4 18:43:31 2013 (r42829) +++ head/en_US.ISO8859-1/htdocs/robots.txt Fri Oct 4 18:55:55 2013 (r42830) @@ -1,7 +1,7 @@ # $FreeBSD$ User-agent: * - +#Crawl-delay: 10 Disallow: /cgi/ Disallow: /statistic Disallow: /sites/downloads.sourceforge.net/
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?201310041855.r94IttLG082111>