Date: Wed, 22 Jul 1998 20:13:08 +0200 From: Wolfram Schneider <wosch@cs.tu-berlin.de> To: www@FreeBSD.ORG Cc: dg@FreeBSD.ORG Subject: Blocked web robots Message-ID: <19980722201308.B4687@cs.tu-berlin.de>
next in thread | raw e-mail | index | archive | help
FYI, I'm now blocking the user agents ^(Teleport|Checkbot|Wget|Web|Digimarc) if they try to access any file under http://www.freebsd.org/cgi Some days ago a Teleport agent downloaded 5 url.cgi scripts per second at freefall and raised the load average up to 12! The robots will get a '403 Forbidden' message. -- Wolfram Schneider <wosch@freebsd.org> http://freebsd.org/~w/
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?19980722201308.B4687>