Date: Sat, 1 Jun 2013 01:03:27 +0200 From: Eitan Adler <eadler@freebsd.org> To: Glen Barber <gjb@freebsd.org> Cc: svn-doc-head@freebsd.org, svn-doc-all@freebsd.org, doc-committers@freebsd.org Subject: Re: svn commit: r41802 - head/en_US.ISO8859-1/htdocs Message-ID: <CAF6rxgne8QPgqqoBcPuk=jDeGXKi7ScGxcMxpdQGUTD3gxhXGg@mail.gmail.com> In-Reply-To: <20130531225923.GA1619@glenbarber.us> References: <201305312254.r4VMsjo0003070@svn.freebsd.org> <20130531225923.GA1619@glenbarber.us>
next in thread | previous in thread | raw e-mail | index | archive | help
On 1 June 2013 00:59, Glen Barber <gjb@freebsd.org> wrote: > On Fri, May 31, 2013 at 10:54:45PM +0000, Eitan Adler wrote: >> Author: eadler >> Date: Fri May 31 22:54:45 2013 >> New Revision: 41802 >> URL: http://svnweb.freebsd.org/changeset/doc/41802 >> >> Log: >> There appear to be configuration errors causing an absurd amount of links to be generated. >> For now, block access to search engines until the problem is properly resolved. >> >> While here, add $FreeBSD$ >> >> Modified: >> head/en_US.ISO8859-1/htdocs/robots.txt >> >> Modified: head/en_US.ISO8859-1/htdocs/robots.txt >> ============================================================================== >> --- head/en_US.ISO8859-1/htdocs/robots.txt Fri May 31 19:10:06 2013 (r41801) >> +++ head/en_US.ISO8859-1/htdocs/robots.txt Fri May 31 22:54:45 2013 (r41802) >> @@ -1,4 +1,8 @@ >> +# $FreeBSD$ >> + >> User-agent: * >> >> Disallow: /cgi/ >> Disallow: /statistic >> +Disallow: /publish.html/hu-hu/ > > This is wrong. > > http://www.freebsd.org/index.html/omgwtfbbqhax I was fixing a specific instance reported by automated tools run by a search engine. A more general fix needs to implemented by clusteradm@. >> +Disallow: /cgi/ > > This already exists in the file... ... Interesting, I am still getting reports of these URLs being indexed. I'll look into that. -- Eitan Adler Source, Ports, Doc committer Bugmeister, Ports Security teams
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAF6rxgne8QPgqqoBcPuk=jDeGXKi7ScGxcMxpdQGUTD3gxhXGg>