From owner-freebsd-www  Sat Jul 19 14:57:59 1997
Return-Path: <owner-freebsd-www>
Received: (from root@localhost)
          by hub.freebsd.org (8.8.5/8.8.5) id OAA13932
          for www-outgoing; Sat, 19 Jul 1997 14:57:59 -0700 (PDT)
Received: from asis.com (root@red.asis.com [206.99.112.12])
          by hub.freebsd.org (8.8.5/8.8.5) with ESMTP id OAA13927
          for <www@freebsd.org>; Sat, 19 Jul 1997 14:57:57 -0700 (PDT)
Received: from megan (ppp-14.asis.com [206.99.112.30]) by asis.com (8.8.5/8.7.3) with SMTP id OAA01286 for <www@freebsd.org>; Sat, 19 Jul 1997 14:57:28 -0700 (PDT)
Message-Id: <3.0.32.19970719145343.00827430@asis.com>
X-Sender: meganm@asis.com
X-Mailer: Windows Eudora Pro Version 3.0 (32)
Date: Sat, 19 Jul 1997 14:53:45 -0700
To: www@freebsd.org
From: Megan McCormack <meganm@asis.com>
Subject: Re: /robots.txt suggestions for mirror sites
Mime-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Sender: owner-freebsd-www@freebsd.org
X-Loop: FreeBSD.org
Precedence: bulk


>Some robots read the mirrored web statistic of www.freebsd.org
>and try all links. This cannot work for local URLs, e.g. /~fsmp
>which only exist on www.freebsd.org. The robots increase 
>the error_log file ;-(
>
>You may add the following two lines to your /robots.txt
>
>	Disallow: /stats-img.html
>	Disallow: /stats.html
>

I'll do this if no one sees any reason not to. 
Megan