From owner-freebsd-questions@FreeBSD.ORG Fri May 4 11:26:24 2007 Return-Path: X-Original-To: freebsd-questions@freebsd.org Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 274F116A402; Fri, 4 May 2007 11:26:24 +0000 (UTC) (envelope-from o.greve@axis.nl) Received: from smtp.interstroom.nl (smtp1.interstroom.nl [80.85.129.3]) by mx1.freebsd.org (Postfix) with ESMTP id DC58313C4C3; Fri, 4 May 2007 11:26:23 +0000 (UTC) (envelope-from o.greve@axis.nl) Received: from 5571fe44.ftth.concepts.nl ([85.113.254.68]:49623 helo=[192.168.1.102]) by smtp.interstroom.nl with esmtpsa (TLS-1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.63) (envelope-from ) id 1HjvvM-0006HV-DP; Fri, 04 May 2007 13:26:19 +0200 In-Reply-To: <463B1499.40102@gmx.de> References: <2BEB30C2-C9C5-43AB-9DCA-5C9A1B0AC2C0@axis.nl> <8DDF332E-A03A-44DC-A87B-D64EC6B91E5A@axis.nl> <1896C639-2518-484B-8CD1-5936811AB093@axis.nl> <463B1499.40102@gmx.de> Mime-Version: 1.0 (Apple Message framework v752.3) Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed Message-Id: Content-Transfer-Encoding: 7bit From: Olaf Greve Date: Fri, 4 May 2007 13:26:07 +0200 To: "[LoN]Kamikaze" X-Mailer: Apple Mail (2.752.3) Cc: Ivan Voras , freebsd-questions@freebsd.org, freebsd-amd64@freebsd.org Subject: Re: How to make Apache (2.2.4) less greedy, or Sendmail less polite? X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 04 May 2007 11:26:24 -0000 Hi, >It might as well be a search engine spider. Perhaps it is... By just tailing the httpd-access.log file a few times over the past 10 minutes or so, at least I already just came across: 74.6.70.45 - - [04/May/2007:13:12:34 +0200] "GET /olympus/tope/ tope_show_entry.php?event=13&pic=1 HTTP/1.0" 200 3209 "-" "Mozilla/ 5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/ slurp)" [...] 66.249.66.176 - - [04/May/2007:13:15:46 +0200] "GET /olympus/tope/ tope_show_entry.php?pic=22&event=7 HTTP/1.1" 200 3792 "-" "Mozilla/ 5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" 65.55.210.7 - - [04/May/2007:13:15:52 +0200] "GET /clomid HTTP/1.0" 404 345 "-" "msnbot/1.0 (+http://search.msn.com/msnbot.htm)" So, there ya go: it looks like Yahoo!, Google AND MSNsearch are battering my server pretty much all at once (though I don't see too many traces of either of them, so perhaps they're not all too hard on the machine)... Hmmm, perhaps time for a "robots.txt" file. Then, doing just some more tailing on the access log shows that the actual load is spread over several sites, with about 4 of the top sites being called from a variety of browsers from all sorts of IP addresses. Another assumption then becomes that perhaps as soon as the USA "wakes up", the sites simply get a lot of real visitors too, hence causing a (legit) high load... Still, this is precisely why I'd like to see which scripts are causing heavy load, such that I can perhaps better tune them. Cheers, Olafo