Date: Fri, 4 May 2007 13:26:07 +0200 From: Olaf Greve <o.greve@axis.nl> To: "[LoN]Kamikaze" <LoN_Kamikaze@gmx.de> Cc: Ivan Voras <ivoras@fer.hr>, freebsd-questions@freebsd.org, freebsd-amd64@freebsd.org Subject: Re: How to make Apache (2.2.4) less greedy, or Sendmail less polite? Message-ID: <D839EED8-655D-404B-84F1-B0DEACEDD014@axis.nl> In-Reply-To: <463B1499.40102@gmx.de> References: <2BEB30C2-C9C5-43AB-9DCA-5C9A1B0AC2C0@axis.nl> <f1eu0a$3h5$1@sea.gmane.org> <8DDF332E-A03A-44DC-A87B-D64EC6B91E5A@axis.nl> <f1f18e$ehm$1@sea.gmane.org> <1896C639-2518-484B-8CD1-5936811AB093@axis.nl> <463B1499.40102@gmx.de>
next in thread | previous in thread | raw e-mail | index | archive | help
Hi, >It might as well be a search engine spider. Perhaps it is... By just tailing the httpd-access.log file a few times over the past 10 minutes or so, at least I already just came across: 74.6.70.45 - - [04/May/2007:13:12:34 +0200] "GET /olympus/tope/ tope_show_entry.php?event=13&pic=1 HTTP/1.0" 200 3209 "-" "Mozilla/ 5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/ slurp)" [...] 66.249.66.176 - - [04/May/2007:13:15:46 +0200] "GET /olympus/tope/ tope_show_entry.php?pic=22&event=7 HTTP/1.1" 200 3792 "-" "Mozilla/ 5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" 65.55.210.7 - - [04/May/2007:13:15:52 +0200] "GET /clomid HTTP/1.0" 404 345 "-" "msnbot/1.0 (+http://search.msn.com/msnbot.htm)" So, there ya go: it looks like Yahoo!, Google AND MSNsearch are battering my server pretty much all at once (though I don't see too many traces of either of them, so perhaps they're not all too hard on the machine)... Hmmm, perhaps time for a "robots.txt" file. Then, doing just some more tailing on the access log shows that the actual load is spread over several sites, with about 4 of the top sites being called from a variety of browsers from all sorts of IP addresses. Another assumption then becomes that perhaps as soon as the USA "wakes up", the sites simply get a lot of real visitors too, hence causing a (legit) high load... Still, this is precisely why I'd like to see which scripts are causing heavy load, such that I can perhaps better tune them. Cheers, Olafo
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?D839EED8-655D-404B-84F1-B0DEACEDD014>