Date: Sat, 19 Jul 1997 14:52:38 -0700 From: Megan McCormack <meganm@asis.com> To: www@freebsd.org Subject: Re: /robots.txt suggestions for mirror sites Message-ID: <3.0.32.19970719145222.00812e30@asis.com>
next in thread | raw e-mail | index | archive | help
At 10:29 PM 7/19/97 +0200, you wrote: > >Some robots read the mirrored web statistic of www.freebsd.org >and try all links. This cannot work for local URLs, e.g. /~fsmp >which only exist on www.freebsd.org. The robots increase >the error_log file ;-( > >You may add the following two lines to your /robots.txt > > Disallow: /stats-img.html > Disallow: /stats.html > I'll do this if no one sees any reason not to. Megan
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?3.0.32.19970719145222.00812e30>