Date: Fri, 25 Jul 1997 10:54:08 +0200 From: Stefan Bethke <stefan@promo.de> To: Megan McCormack <meganm@asis.com>, www@FreeBSD.ORG Subject: Re: robots.txt Message-ID: <l03102801affe19c9821a@[194.45.188.81]> In-Reply-To: <3.0.32.19970724213535.008c8310@asis.com>
next in thread | previous in thread | raw e-mail | index | archive | help
At 6:35 Uhr +0200 25.07.1997, Megan McCormack wrote: >Hello, > >Does it make sense to disallow stats.html in robots.txt >so that mirror sites error logs don't overflow with >error messages of robots trying to scoop up the links to >local pages like ~jkh on the mirrors? No, rather make analog produce absolute URLs. Btw, /robots.txt currently reads User-agent: * Disallow: /cgi-bin/ Disallow: /cgi/ Disallow: ~/jfieber/data/cgi/ it should be ... Disallow: /~jfieber/data/cgi/ Stefan -- Stefan Bethke Promo Datentechnik | Tel. +49-40-851744-0 + Systemberatung GmbH | Fax. +49-40-851744-44 Eduardstrasse 46-48 | e-mail: stefan@Promo.DE D-20257 Hamburg | http://www.Promo.DE/
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?l03102801affe19c9821a>