Date: Sun, 15 Jun 2008 18:18:12 +0100 From: Stut <stuttle@gmail.com> To: freebsd-questions@freebsd.org Subject: High load, lots of free memory and processes in devfs state Message-ID: <CD3881B1-2A55-48A4-907F-986017DDC09B@gmail.com>
next in thread | raw e-mail | index | archive | help
Hi, I'm having a serious problem with one of my FreeBSD servers. It runs FreeBSD 6.2-RELEASE, Apache 2.2.8 and PHP 5.2.1. I've checked the hardware and it's all OK. There's approximately 400 Apache processes running and a 2GB memcached instance. Up until last weekend this server was working perfectly at a level of HTTP traffic higher than it's currently getting. Last week our database server (separate server) died and had to be rebuilt. While this was being done this server hosted the database. This has now been completed and the PHP app is pointing back at the dedicated DB server. Top shows the following... last pid: 26838; load averages: 10.22, 14.06, 13.55 up 2+00:34:47 18:03:43 619 processes: 1 running, 618 sleeping CPU states: 4.9% user, 0.0% nice, 24.8% system, 0.4% interrupt, 70.0% idle Mem: 2241M Active, 2718M Inact, 462M Wired, 394M Cache, 214M Buf, 1747M Free Swap: 8192M Total, 124K Used, 8192M Free PID UID THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 26807 80 1 -4 0 79892K 9996K devfs 0 0:00 2.81% httpd 26797 80 1 -4 0 82376K 12592K devfs 1 0:00 2.19% httpd 26791 80 1 -4 0 82376K 12636K devfs 0 0:00 1.85% httpd 26783 80 1 -4 0 82392K 12640K devfs 3 0:00 1.84% httpd 26780 80 1 -4 0 79892K 10036K devfs 0 0:00 1.51% httpd 26801 80 1 -4 0 81292K 11104K devfs 3 0:00 1.50% httpd 26786 80 1 -4 0 80456K 10796K devfs 1 0:00 1.41% httpd 26784 80 1 -4 0 79892K 10036K devfs 0 0:00 1.41% httpd 26785 80 1 -4 0 81304K 11228K devfs 0 0:00 1.41% httpd 26763 80 1 -4 0 83220K 13752K devfs 1 0:00 1.24% httpd ...etc... As you can see there's plenty of free memory and the CPU is 70% idle yet the load is sky high. When it's like this it's impossible to do anything - every command can take anything from a few seconds to a few minutes to respond - and the web user experience is shot to pieces. I can't find a reference that explains what the devfs state indicates. SA shows the following... root@harold:~# sa 150107 242633.24re 1548.32cp 13avio 801k 31174 192785.02re 1488.13cp 32avio 502k httpd* 7047 6286.71re 21.53cp 67avio 2854k php 7703 164.83re 14.65cp 3avio 743k convert 8 19.82re 9.47cp 5016avio 1k gzip 7703 177.66re 7.70cp 2avio 1352k composite 4 18.90re 3.69cp 1avio 3k mysqldump 9 0.80re 0.76cp 23avio 144k bzip2 6 4.16re 0.53cp 22998avio 1731k find 2890 1126.31re 0.45cp 0avio 175988k perl* 5415 40.83re 0.32cp 11avio 123676k rateup 3009 14948.72re 0.32cp 3avio 9512k smtp 247 1812.85re 0.27cp 440avio 945k cleanup 7 163.69re 0.18cp 0avio 105k top 167 1809.71re 0.11cp 183avio 1117k pickup 2667 797.34re 0.07cp 0avio 29671k trivial- rewrite 1820 96.40re 0.02cp 0avio 141901k ps 17 1482.98re 0.02cp 1avio 1083k sshd* 35629 6318.10re 0.01cp 0avio 18016816k sh 15232 391.68re 0.01cp 0avio 935430k sendmail 599 1223.30re 0.01cp 28avio 44065k bounce 7 0.95re 0.01cp 8avio 596k sa 35 1952.66re 0.01cp 11avio 16013k ***other 25 0.07re 0.01cp 18avio 1447k rm 15232 203.94re 0.00cp 7avio 2994436k postdrop 61 1.20re 0.00cp 5avio 34481k git 3 123.65re 0.00cp 12avio 796k mail 92 345.77re 0.00cp 0avio 34405k scache 31 2.13re 0.00cp 66avio 7115k newsyslog 6 2140.32re 0.00cp 14avio 6835k bash 5463 290.01re 0.00cp 0avio 2584397k grep 5 1479.06re 0.00cp 17avio 3878k sshd 1830 99.75re 0.00cp 0avio 1221440k wc 4 2.86re 0.00cp 2avio 13472k less 3 0.46re 0.00cp 0avio 10048k pgrep 68 27.44re 0.00cp 0avio 69632k sh* 3 5.06re 0.00cp 1avio 13120k showq 59 109.55re 0.00cp 0avio 243008k smtpd 183 0.54re 0.00cp 2avio 518400k unlink 2 1.61re 0.00cp 33avio 4608k whereis 12 0.00re 0.00cp 0avio 35000k adjkerntz 50 189.34re 0.00cp 0avio 165000k anvil 364 1.27re 0.00cp 0avio 947300k atrun 3 0.00re 0.00cp 0avio 53100k awk 10 0.05re 0.00cp 0avio 298800k bash* 27 4.15re 0.00cp 0avio 62000k cat 7 0.00re 0.00cp 0avio 44800k cmp 40 0.01re 0.00cp 2avio 79100k cp 2937 5227.93re 0.00cp 0avio 5143200k cron* 6 0.00re 0.00cp 0avio 22800k date 183 14.68re 0.00cp 2avio 578900k dd 5 0.00re 0.00cp 0avio 10600k egrep 132 241.91re 0.00cp 0avio 552400k error 6 0.05re 0.00cp 0avio 0k expr 245 444.67re 0.00cp 1avio 1181500k flush 3 0.00re 0.00cp 0avio 0k hostname 8 0.03re 0.00cp 0avio 21000k id 4 0.04re 0.00cp 0avio 173800k ipcs 183 0.49re 0.00cp 0avio 670000k jot 16 0.35re 0.00cp 1avio 65900k ls 3 1.52re 0.00cp 11avio 16300k man 2 0.06re 0.00cp 3avio 6800k manpath 10 0.00re 0.00cp 2avio 29200k mktemp 2 0.00re 0.00cp 0avio 0k mount 1287 4.64re 0.00cp 3avio 3717800k mv 4 0.12re 0.00cp 0avio 17600k postqueue 57 10.01re 0.00cp 0avio 242300k proxymap 7 4.19re 0.00cp 0avio 60500k sed 7 4.18re 0.00cp 0avio 10100k sort 7 0.05re 0.00cp 0avio 6100k sysctl 9 0.00re 0.00cp 0avio 15000k tee 4 26.15re 0.00cp 0avio 9300k time 2 0.48re 0.00cp 0avio 25400k zcat I'm fairly new to SA but I can't see anything in there that indicates a problem. If I'm missing something please let mw know. If anyone has any ideas as to what might be causing this and how to fix it I'd appreciate it. I'm getting close to giving up and simply reinstalling it. -Stut -- http://stut.net/
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CD3881B1-2A55-48A4-907F-986017DDC09B>