From owner-freebsd-www Wed Jul 22 11:18:19 1998 Return-Path: Received: (from daemon@localhost) by hub.freebsd.org (8.8.8/8.8.8) id LAA22055 for www-outgoing; Wed, 22 Jul 1998 11:18:19 -0700 (PDT) (envelope-from owner-freebsd-www) Received: from mail.cs.tu-berlin.de (root@mail.cs.tu-berlin.de [130.149.17.13]) by hub.freebsd.org (8.8.8/8.8.8) with ESMTP id LAA21982; Wed, 22 Jul 1998 11:18:06 -0700 (PDT) (envelope-from wosch@cs.tu-berlin.de) Received: from caramba.cs.tu-berlin.de (wosch@caramba.cs.tu-berlin.de [130.149.17.12]) by mail.cs.tu-berlin.de (8.9.1/8.9.1) with ESMTP id UAA09241; Wed, 22 Jul 1998 20:13:11 +0200 (MET DST) Received: (from wosch@localhost) by caramba.cs.tu-berlin.de (8.9.1/8.9.0) id UAA04941; Wed, 22 Jul 1998 20:13:08 +0200 (MET DST) Message-ID: <19980722201308.B4687@cs.tu-berlin.de> Date: Wed, 22 Jul 1998 20:13:08 +0200 From: Wolfram Schneider To: www@FreeBSD.ORG Cc: dg@FreeBSD.ORG Subject: Blocked web robots Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Mailer: Mutt 0.93.1i Sender: owner-freebsd-www@FreeBSD.ORG X-Loop: FreeBSD.org Precedence: bulk FYI, I'm now blocking the user agents ^(Teleport|Checkbot|Wget|Web|Digimarc) if they try to access any file under http://www.freebsd.org/cgi Some days ago a Teleport agent downloaded 5 url.cgi scripts per second at freefall and raised the load average up to 12! The robots will get a '403 Forbidden' message. -- Wolfram Schneider http://freebsd.org/~w/