From owner-freebsd-questions@FreeBSD.ORG Sat Nov 6 05:42:07 2004 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 2540916A4CE for ; Sat, 6 Nov 2004 05:42:07 +0000 (GMT) Received: from inertia.drifthost.com (inertial.drifthost.com [66.90.101.6]) by mx1.FreeBSD.org (Postfix) with ESMTP id 82A8143D86 for ; Sat, 6 Nov 2004 05:42:06 +0000 (GMT) (envelope-from steve@drifthost.com) Received: from [203.142.133.217] (helo=Steve) by inertia.drifthost.com with esmtpa (Exim 4.42 (FreeBSD)) id 1CQJKt-000EJm-KK for questions@freebsd.org; Sat, 06 Nov 2004 16:42:09 +1100 From: "Steven Adams" To: Date: Sat, 6 Nov 2004 16:41:50 +1100 MIME-Version: 1.0 X-Mailer: Microsoft Office Outlook, Build 11.0.5510 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.2180 Thread-Index: AcTDw1BL5UluTySnRvudT/hW3OANSA== X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - inertia.drifthost.com X-AntiAbuse: Original Domain - freebsd.org X-AntiAbuse: Originator/Caller UID/GID - [0 0] / [26 6] X-AntiAbuse: Sender Address Domain - drifthost.com X-Source: X-Source-Args: X-Source-Dir: Message-Id: <20041106054206.82A8143D86@mx1.FreeBSD.org> Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.1 Subject: Big Problem, Load Avg Very High X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: drift@FreeBSD.ORG List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 06 Nov 2004 05:42:07 -0000 Hi, I run FreeBSD 5.2.1 with the following hardware Acer Server: Altos LSI Megaraid with 5x 36 gig scsi on raid 5. Dual xeon cpu's 1GIG ECC Ram 2x 250GIG IDE Drives We host a couple or sites on this server (not very big sites). My server load is always around 0.90 - 3.40. Sometimes it will jump up to 10-15. At random it will jump up to 30-40 load and I wont even be able to get to the server, typing commands on the remote ip based kvm is VERY slow sometimes missing letters. As soon as im able to get top running it shows ==================================== last pid: 52614; load averages: 6.82, 15.75, 15.18 up 1+03:07:12 16:32:22 462 processes: 1 running, 460 sleeping, 1 zombie CPU states: 0.0% user, 0.0% nice, 0.6% system, 0.6% interrupt, 98.7% idle Mem: 615M Active, 68M Inact, 288M Wired, 29M Cache, 112M Buf, 1844K Free Swap: 1536M Total, 555M Used, 981M Free, 36% Inuse, 12K In =================================== I am trying to figure out why the load is so high for a server that should be able to handle it easily.. I did notice once when running systat -vmstat the amr0(scsi raid) jumps up to 99% busy copying 2-3MB/s for a few moments then goes back down.. After one of the times it went to 50load it got this on the console screen. FYI: amrd0s1h is /home partition ================================================== Swap_pager: indefinite wait buffer: device: amrd0s1h, blkno: 103776, size: 32768 Swap_pager: indefinite wait buffer: device: amrd0s1h, blkno: 130801, size: 4096 ================================================== I wanted to check the logs but for some weird reason syslogd is not logging to /var/log/security and /var/log/blah I tried to do a killall -HUP syslog and killall -9 syslogd ; syslogd It logs to the other log files but wont log to the /var/log/blah and /var/log/security so I cant see if there Is a problem with the server. My config is below. ================================================================ *.* /var/log/blah *.err;kern.debug;auth.notice;mail.crit /dev/console *.notice;authpriv.none;kern.debug;lpr.info;mail.crit;news.err /var/log/messages security.* /var/log/security auth.info;authpriv.info /var/log/auth.log local0.notice;local0.debug;mail.*;mail.none;mail.info;local0.info /var/log/maillog lpr.info /var/log/lpd-errs ftp.info /var/log/xferlog cron.* /var/log/cron *.=debug /var/log/debug.log *.emerg * # uncomment this to log all writes to /dev/console to /var/log/console.log #console.info /var/log/console.log # uncomment this to enable logging of all log messages to /var/log/all.log # touch /var/log/all.log and chmod it to mode 600 before it will work #*.* /var/log/all.log # uncomment this to enable logging to a remote loghost named loghost #*.* @loghost # uncomment these if you're running inn # news.crit /var/log/news/news.crit # news.err /var/log/news/news.err # news.notice /var/log/news/news.notice !startslip *.* /var/log/slip.log !ppp *.* /var/log/ppp.log ================================================================ I am really confussed what else I can do to find out what is causing this problem as the server completely locks up when it goes to 50+ load. It seems that its copying a lot of information to the swap drive and is running out of ram, which I don't know why it seems apache is taking up all of the ram for some weird reason? Here is my demsg, I hope you guys can help! ============================================== Copyright (c) 1992-2004 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD 5.2.1-RELEASE #3: Wed Sep 29 20:50:16 EST 2004 root@inertia.drifthost.com:/usr/src/sys/i386/compile/DRIFTHOST Preloaded elf kernel "/boot/kernel/kernel" at 0xc070a000. Preloaded elf module "/boot/kernel/acpi.ko" at 0xc070a2bc. ACPI APIC Table: Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: Intel(R) Xeon(TM) CPU 2.40GHz (2392.05-MHz 686-class CPU) Origin = "GenuineIntel" Id = 0xf29 Stepping = 9 Features=0xbfebfbff Hyperthreading: 2 logical CPUs real memory = 1073676288 (1023 MB) avail memory = 1037717504 (989 MB) FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 cpu2 (AP): APIC ID: 2 cpu3 (AP): APIC ID: 3 ioapic0 irqs 0-15 on motherboard ioapic1 irqs 16-31 on motherboard ioapic2 irqs 32-47 on motherboard Pentium Pro MTRR support enabled ACPI-0660: *** Warning: Type override - [DEB_] had invalid type (Integer) for Scope operator, changed to (Scope) ACPI-0660: *** Warning: Type override - [MLIB] had invalid type (Integer) for Scope operator, changed to (Scope) ACPI-0660: *** Warning: Type override - [IO__] had invalid type (Integer) for Scope operator, changed to (Scope) ACPI-0660: *** Warning: Type override - [DATA] had invalid type (String) for Scope operator, changed to (Scope) ACPI-0660: *** Warning: Type override - [SIO_] had invalid type (String) for Scope operator, changed to (Scope) ACPI-0660: *** Warning: Type override - [SB__] had invalid type (String) for Scope operator, changed to (Scope) ACPI-0660: *** Warning: Type override - [PM__] had invalid type (String) for Scope operator, changed to (Scope) ACPI-0660: *** Warning: Type override - [ICNT] had invalid type (String) for Scope operator, changed to (Scope) ACPI-0660: *** Warning: Type override - [ACPI] had invalid type (String) for Scope operator, changed to (Scope) ACPI-0660: *** Warning: Type override - [IORG] had invalid type (String) for Scope operator, changed to (Scope) ACPI-0660: *** Warning: Type override - [SB__] had invalid type (String) for Scope operator, changed to (Scope) ACPI-0660: *** Warning: Type override - [PM__] had invalid type (String) for Scope operator, changed to (Scope) ACPI-0660: *** Warning: Type override - [SIO_] had invalid type (String) for Scope operator, changed to (Scope) ACPI-0660: *** Warning: Type override - [PM__] had invalid type (String) for Scope operator, changed to (Scope) ACPI-0660: *** Warning: Type override - [BIOS] had invalid type (Integer) for Scope operator, changed to (Scope) ACPI-0660: *** Warning: Type override - [CMOS] had invalid type (Integer) for Scope operator, changed to (Scope) ACPI-0660: *** Warning: Type override - [KBC_] had invalid type (Integer) for Scope operator, changed to (Scope) ACPI-0660: *** Warning: Type override - [OEM_] had invalid type (Integer) for Scope operator, changed to (Scope) npx0: [FAST] npx0: on motherboard npx0: INT 16 interface acpi0: on motherboard Timecounter "ACPI-safe" frequency 3579545 Hz quality 1000 pcibios: BIOS version 2.10 Using $PIR table, 10 entries at 0xc00f4b00 acpi_timer0: <32-bit timer at 3.579545MHz> port 0x508-0x50b on acpi0 acpi_cpu0: on acpi0 acpi_cpu1: on acpi0 acpi_cpu2: on acpi0 acpi_cpu3: on acpi0 pcib0: port 0xcf8-0xcff on acpi0 pci0: on pcib0 bge0: mem 0xfe9e0000-0xfe9effff irq 16 at device 2.0 on pci0 bge0: Ethernet address: 00:c0:9f:25:6a:ad miibus0: on bge0 brgphy0: on miibus0 brgphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseTX, 1000baseTX-FDX, auto pci0: at device 14.0 (no driver attached) isab0: at device 15.0 on pci0 isa0: on isab0 atapci0: port 0xffa0-0xffaf,0x374-0x377,0x170-0x177,0x3f4-0x3f7,0x1f0-0x1f7 at device 0 ata0: at 0x1f0 irq 14 on atapci0 ata0: [MPSAFE] ata1: at 0x170 irq 15 on atapci0 ata1: [MPSAFE] pcib1: on acpi0 pci1: on pcib1 pcib2: on acpi0 pci2: on pcib2 amr0: mem 0xfc8f0000-0xfc8fffff irq 24 at device 4.0 on pci2 amr0: Firmware 1L26, BIOS G112, 64MB RAM acpi_button0: on acpi0 atkbdc0: port 0x64,0x60 irq 1 on acpi0 atkbd0: flags 0x1 irq 1 on atkbdc0 kbd0 at atkbd0 psm0: irq 12 on atkbdc0 psm0: model Generic PS/2 mouse, device ID 0 fdc0: ready for input in output fdc0: cmd 3 failed at out byte 1 of 3 fdc0: ready for input in output fdc0: cmd 3 failed at out byte 1 of 3 orm0: