Date: Thu, 29 Jan 2009 16:51:27 +0300 From: Artem Kuchin <matrix@itlegion.ru> To: Bill Moran <wmoran@potentialtech.com> Cc: "freebsd-questions@freebsd.org" <freebsd-questions@freebsd.org> Subject: Re: 6.8 became very slow Message-ID: <4981B45F.8040606@itlegion.ru> In-Reply-To: <20090129082654.f4f9ae25.wmoran@potentialtech.com> References: <498196B8.1060101@itlegion.ru> <20090129082654.f4f9ae25.wmoran@potentialtech.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Bill Moran ?????: > In response to Artem Kuchin <matrix@itlegion.ru>: > > >> I have a very strange situation here. There was a hosting box with 5 >> jails. >> Everything is 6.4 >> It was running twe driver with RAID 5. >> >> Then i had a crash and had to reinstall the system. >> >> So, i have installed FREEBSD 6.8, cvsed the latest, rebuilt everything >> and then >> just copied jails from the prev installation. So, the host system is 6.8 >> and the jails are 6.4. >> Also, raid is MIRROR now, not RAID5. >> > > OK. I originally thought your subject was a typo, but this is the > second place where you mentioned 6.8. > > Not only would you have to go into the future to get 6.8, but you'd > have to slip into an alternate reality, since 6.5 is expected to be > the last release on the 6 branch. > > What is the output of uname -a? If you really have something called > 6.8, where on earth did you get it? > > Apparently i just made it up. I should change the tea i drink or check the water source. Of course the new is still 6.4 just updates to the latest sources. Maybe i got confuses about the version since it is the last server running version 6 and i was kind of subconsciously sure that 6.8 must be the last before 7 :) >> The problem is that now everything became really slow. >> >> When i do top i see: >> load averages: 16.45, 14.86, 13.86 >> 737 processes: 20 running, 703 sleeping, 14 zombie >> CPU: 15.9% user, 0.0% nice, 81.9% system, 2.2% interrupt, 0.0% idle >> >> Everything pretty much as it was. But 82% system CPU is really weird. I >> don;t remember >> exactly, but i think it was not like this before. AFAIK it meas 82% of >> CPU time is spent in >> the kernel. >> > > gstat output seems to indicate nothing unusual, so the time spent in > the kernel must be for something else, network traffic maybe? Try > checking top with -m io to see if anything is showing an unusually > high # of context switches. > > Perhaps do a little easter egg hunting and try shutting down processes > to see what is using up all the system time. Once you've got it > narrowed down you can run ktrace on the problematic process to see > what it's doing. > > Well, the trick is it was not so before and i did not change anything. It is a web hosting server and host around 100 sites with a lot of perl running. 22533 xxx 1 130 0 21276K 12904K CPU2 2 0:02 62.68% perl5.8.8 22545 xxx 1 116 0 3592K 2992K RUN 3 0:01 51.00% perl 22542 xxx 1 -4 0 5448K 3908K CPU0 0 0:01 43.05% perl 22546 xxx 1 -4 0 3564K 2968K RUN 0 0:01 36.00% perl 22501 xxx 1 4 0 25168K 17148K sbwait 3 0:04 20.84% perl5.8.8 21165 xxx 1 130 0 15168K 12248K RUN 0 0:59 17.29% perl 22556 xxx 1 -4 0 3596K 2900K RUN 0 0:00 15.00% perl5.8.8 18125 xxx 1 113 0 26912K 18908K select 2 0:10 14.94% httpd 22495 xxx 1 4 0 25168K 17152K sbwait 0 0:04 12.50% perl5.8.8 22496 xxx 1 4 0 15056K 13832K sbwait 3 0:04 12.01% perl5.8.8 22502 xxx 1 4 0 8640K 7436K sbwait 1 0:02 11.57% perl5.8.8 It they have been running before and it was a lot faster. top -m io shows 0.00% for all processes. The memory is more than enough: Mem: 3081M Active, 3702M Inact, 339M Wired, 395M Cache, 112M Buf, 511M Free I tried disabling mysql just for fun. I did not help. Shutting down process by process is not an option because 1) it is a production server 2) there over 700 processes trafshow shows average CPS about 400K/sec - not much and as it was before. So, is there anyway to see what kernel is doing and why system load is so high? -- Artem
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4981B45F.8040606>