Date: Fri, 10 Nov 1995 03:23:12 +0100 (MET) From: Piero Serini <piero@strider.ibenet.it> To: gcrutcher@datatrek.com (Gary Crutcher) Cc: hackers@freebsd.org Subject: Re: robot.txt file Message-ID: <199511100223.DAA29839@strider.ibenet.it> In-Reply-To: <199511100150.RAA11236@dtihost.datatrek.com> from "Gary Crutcher" at Nov 9, 95 05:50:44 pm
next in thread | previous in thread | raw e-mail | index | archive | help
Hello. Quoting from Gary Crutcher (Fri Nov 10 02:50:44 1995): > I have been checking my apache server's logfile and noticed that a number > sites have been trying to access a file called "robot.txt". What is this > file and do I need it on my system. The file /robot.txt, if present, tells the robots if they are welcome to your site or not, and if they are tells them which directory they cannot access, if any. For more informations, check: http://info.webcrawler.com/mak/projects/robots/robots.html I find the whole very intersting, but if you only want to exclude robots, follow the link "A standard for Robot exclusion". Bye, -- # $Id: .signature,v 1.12 1995/08/14 12:10:54 piero Exp $ Piero Serini Via Giambologna, 1 <Piero@Free.IT> I 20136 Milano - ITALY
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?199511100223.DAA29839>