Date: Sat, 19 Jul 1997 22:29:50 +0200 (MET DST) From: Wolfram Schneider <wosch@apfel.de> To: www@freebsd.org Subject: /robots.txt suggestions for mirror sites Message-ID: <199707192029.WAA05178@panke.panke.de>
next in thread | raw e-mail | index | archive | help
Some robots read the mirrored web statistic of www.freebsd.org and try all links. This cannot work for local URLs, e.g. /~fsmp which only exist on www.freebsd.org. The robots increase the error_log file ;-( You may add the following two lines to your /robots.txt Disallow: /stats-img.html Disallow: /stats.html -- Wolfram Schneider <wosch@apfel.de> http://www.apfel.de/~wosch/
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?199707192029.WAA05178>