Date: Fri, 4 Oct 2013 18:57:43 +0000 (UTC) From: Glen Barber <gjb@FreeBSD.org> To: doc-committers@freebsd.org, svn-doc-all@freebsd.org, svn-doc-head@freebsd.org Subject: svn commit: r42831 - head/en_US.ISO8859-1/htdocs Message-ID: <201310041857.r94Ivh9I083239@svn.freebsd.org>
next in thread | raw e-mail | index | archive | help
Author: gjb Date: Fri Oct 4 18:57:43 2013 New Revision: 42831 URL: http://svnweb.freebsd.org/changeset/doc/42831 Log: Explicitly list the files within cgi/, since not all crawlers will honor the following syntax: Allow: /cgi/foo Disallow: /cgi/ Modified: head/en_US.ISO8859-1/htdocs/robots.txt Modified: head/en_US.ISO8859-1/htdocs/robots.txt ============================================================================== --- head/en_US.ISO8859-1/htdocs/robots.txt Fri Oct 4 18:55:55 2013 (r42830) +++ head/en_US.ISO8859-1/htdocs/robots.txt Fri Oct 4 18:57:43 2013 (r42831) @@ -2,6 +2,21 @@ User-agent: * #Crawl-delay: 10 -Disallow: /cgi/ +Disallow: /cgi/confirm-code.cgi +Disallow: /cgi/cvsweb.cgi +Disallow: /cgi/dosendpr.cgi +Disallow: /cgi/getmsg.cgi +Disallow: /cgi/mailindex.cgi +Disallow: /cgi/man.cgi +Disallow: /cgi/mid.cgi +Disallow: /cgi/mirror.cgi +Disallow: /cgi/missing_handler.cgi +Disallow: /cgi/monthly.cgi +Disallow: /cgi/pds.cgi +Disallow: /cgi/ports.cgi +Disallow: /cgi/query-pr-summary.cgi +Disallow: /cgi/query-pr.cgi +Disallow: /cgi/search.cgi +Disallow: /cgi/url.cgi Disallow: /statistic Disallow: /sites/downloads.sourceforge.net/
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?201310041857.r94Ivh9I083239>