From owner-freebsd-questions@FreeBSD.ORG Sun May 17 11:32:09 2009 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 72916106566B for ; Sun, 17 May 2009 11:32:09 +0000 (UTC) (envelope-from dgl@dlee.org) Received: from mail.ff44a.com (mail.ff44a.com [64.127.120.100]) by mx1.freebsd.org (Postfix) with ESMTP id 598018FC08 for ; Sun, 17 May 2009 11:32:09 +0000 (UTC) (envelope-from dgl@dlee.org) Received: from localhost (localhost.localdomain [127.0.0.1]) by mail.ff44a.com (Postfix) with ESMTP id 50A7F7E4093; Sun, 17 May 2009 03:57:20 -0700 (PDT) X-Virus-Scanned: amavisd-new at mail.ff44a.com Received: from mail.ff44a.com ([127.0.0.1]) by localhost (mail.ff44a.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id m0NQqenieV3q; Sun, 17 May 2009 03:57:08 -0700 (PDT) Received: from mini.local (pool-71-178-105-137.washdc.east.verizon.net [71.178.105.137]) by mail.ff44a.com (Postfix) with ESMTP id 6D81A7E4091; Sun, 17 May 2009 03:57:07 -0700 (PDT) Date: Sun, 17 May 2009 07:06:57 -0400 From: Doug Lee To: freebsd-questions@freebsd.org Message-ID: <20090517110657.GC2706@mini.local> Mail-Followup-To: Doug Lee , freebsd-questions@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.5.18 (2008-05-17) Subject: 4.11 panic every 23 hours 55 minutes or so X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 17 May 2009 11:32:09 -0000 One of the weirder things I've seen in a while here... OS: FreeBSD 4.11 (yeah I know, old, but generally stable) CPU: Intel(R) Pentium(R) 4 CPU 2.00GHz real memory = 536608768 (524032K bytes) Hds: IDE Problem: Ever since a suspitious power outage (I say suspitious because we think a surge was also involved), this box has been exhibiting kernel panics about every 23 hours 55 minutes, give or take about 4 minutes either way. Obviously hardware is suspect, and hopefully in line for upgrade; but as FreeBSD has always proven so stable for me, I'm curious what on earth could cause this sort of regular panic? It's not time of day; if I reboot at 2:00 AM, 3:55 PM, or any other time, it's 23:55 or so later I get a panic, whenever that may be. I think this rules out cron jobs, external attacks, and load-based issues. I'll provide the latest two panic logs below, followed by a few rounds of output from `top -Sd1000', which is `top' with system processes shown and dumb-terminal output, logged via ssh onto another system so I could see the last report before the panic. I have a longer `top' log but I'll send that only on request to avoid an even huger post. Curiously, this is one of the few times the uptime went past 24 hours, and also shows the current process as something other than Idle. I wonder if running `top' actually changed the results. Any info would be most welcome. Please Cc me on responses. ----- Panic from yesterday (double panic; this is typical) ----- Fatal trap 12: page fault while in kernel mode fault virtual address = 0xe4f5eb02 fault code = supervisor read, page not present instruction pointer = 0x8:0xc02af097 stack pointer = 0x10:0xc04be8d4 frame pointer = 0x10:0xc04be8d8 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = Idle interrupt mask = trap number = 12 panic: page fault syncing disks... Fatal trap 12: page fault while in kernel mode fault virtual address = 0x30 fault code = supervisor read, page not present instruction pointer = 0x8:0xc0377070 stack pointer = 0x10:0xc04be6f4 frame pointer = 0x10:0xc04be6fc code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = Idle interrupt mask = bio trap number = 12 panic: page fault Uptime: 23h59m5s Automatic reboot in 15 seconds - press a key on the console to abort ----- Panic from today (just one this time, NOT typical) ----- Fatal trap 12: page fault while in kernel mode fault virtual address = 0xe694f564 fault code = supervisor read, page not present instruction pointer = 0x8:0xc02a7b73 stack pointer = 0x10:0xdb394da4 frame pointer = 0x10:0xdb394e28 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 505 (nmbd) interrupt mask = none trap number = 12 panic: page fault syncing disks... 1 done Uptime: 1d0h1m36s Automatic reboot in 15 seconds - press a key on the console to abort Rebooting... ----- top -Sd10000 from today (stopping at the above panic) ----- last pid: 9015; load averages: 0.02, 0.02, 0.00 up 1+00:01:56 06:37:45 76 processes: 1 running, 74 sleeping, 1 zombie CPU states: 0.0% user, 0.0% nice, 0.4% system, 0.0% interrupt, 99.6% idle Mem: 95M Active, 264M Inact, 63M Wired, 21M Cache, 60M Buf, 55M Free Swap: 250M Total, 250M Free PID USERNAME PRI NICE SIZE RES STATE TIME WCPU CPU COMMAND 491 mysql 2 0 44788K 19244K poll 0:57 0.00% 0.00% mysqld 9 root 18 0 0K 0K syncer 0:16 0.00% 0.00% syncer 330 ssbdev 2 0 9744K 8188K poll 0:12 0.00% 0.00% python 248 root 2 0 1336K 868K select 0:04 0.00% 0.00% ntpd 224 root 2 0 464K 252K select 0:03 0.00% 0.00% natd 416 root 2 0 3188K 1948K select 0:03 0.00% 0.00% sendmail 6408 root 10 0 2868K 2292K nanslp 0:03 0.00% 0.00% perl5.8.8 308 root 2 0 9884K 5732K select 0:02 0.00% 0.00% httpd 6515 root 10 0 16548K 15872K nanslp 0:02 0.00% 0.00% perl5.8.8 505 root 2 0 5364K 1796K select 0:02 0.00% 0.00% nmbd 6738 root 10 0 16548K 15872K nanslp 0:01 0.00% 0.00% perl5.8.8 650 dlee 2 0 5336K 1836K select 0:01 0.00% 0.00% sshd 7778 root 10 0 16532K 15868K nanslp 0:01 0.00% 0.00% perl5.8.8 8307 root 10 0 16532K 15868K nanslp 0:01 0.00% 0.00% perl5.8.8 245 bind 2 0 2484K 1856K select 0:01 0.00% 0.00% named 241 root 2 0 1000K 668K select 0:01 0.00% 0.00% syslogd 8236 root 10 0 16516K 15856K nanslp 0:01 0.00% 0.00% perl5.8.8 10 root -2 0 0K 0K vlruwt 0:01 0.00% 0.00% vnlru last pid: 9015; load averages: 0.02, 0.02, 0.00 up 1+00:01:58 06:37:47 76 processes: 1 running, 74 sleeping, 1 zombie CPU states: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle Mem: 95M Active, 264M Inact, 63M Wired, 21M Cache, 60M Buf, 55M Free Swap: 250M Total, 250M Free PID USERNAME PRI NICE SIZE RES STATE TIME WCPU CPU COMMAND 491 mysql 2 0 44788K 19244K poll 0:57 0.00% 0.00% mysqld 9 root 18 0 0K 0K syncer 0:16 0.00% 0.00% syncer 330 ssbdev 2 0 9744K 8188K poll 0:12 0.00% 0.00% python 248 root 2 0 1336K 868K select 0:04 0.00% 0.00% ntpd 224 root 2 0 464K 252K select 0:03 0.00% 0.00% natd 416 root 2 0 3188K 1948K select 0:03 0.00% 0.00% sendmail 6408 root 10 0 2868K 2292K nanslp 0:03 0.00% 0.00% perl5.8.8 308 root 2 0 9884K 5732K select 0:02 0.00% 0.00% httpd 6515 root 10 0 16548K 15872K nanslp 0:02 0.00% 0.00% perl5.8.8 505 root 2 0 5364K 1804K select 0:02 0.00% 0.00% nmbd 6738 root 10 0 16548K 15872K nanslp 0:01 0.00% 0.00% perl5.8.8 650 dlee 2 0 5336K 1836K select 0:01 0.00% 0.00% sshd 7778 root 10 0 16532K 15868K nanslp 0:01 0.00% 0.00% perl5.8.8 8307 root 10 0 16532K 15868K nanslp 0:01 0.00% 0.00% perl5.8.8 245 bind 2 0 2484K 1856K select 0:01 0.00% 0.00% named 241 root 2 0 1000K 668K select 0:01 0.00% 0.00% syslogd 8236 root 10 0 16516K 15856K nanslp 0:01 0.00% 0.00% perl5.8.8 10 root -2 0 0K 0K vlruwt 0:01 0.00% 0.00% vnlru last pid: 9015; load averages: 0.02, 0.02, 0.00 up 1+00:02:00 06:37:49 76 processes: 1 running, 74 sleeping, 1 zombie CPU states: 0.4% user, 0.0% nice, 0.4% system, 0.4% interrupt, 98.8% idle Mem: 95M Active, 264M Inact, 63M Wired, 21M Cache, 60M Buf, 55M Free Swap: 250M Total, 250M Free PID USERNAME PRI NICE SIZE RES STATE TIME WCPU CPU COMMAND 491 mysql 2 0 44788K 19244K poll 0:57 0.00% 0.00% mysqld 9 root 18 0 0K 0K syncer 0:16 0.00% 0.00% syncer 330 ssbdev 2 0 9744K 8188K poll 0:12 0.00% 0.00% python 248 root 2 0 1336K 868K select 0:04 0.00% 0.00% ntpd 224 root 2 0 464K 252K select 0:03 0.00% 0.00% natd 416 root 2 0 3188K 1948K select 0:03 0.00% 0.00% sendmail 6408 root 10 0 2868K 2292K nanslp 0:03 0.00% 0.00% perl5.8.8 308 root 2 0 9884K 5732K select 0:02 0.00% 0.00% httpd 6515 root 10 0 16548K 15872K nanslp 0:02 0.00% 0.00% perl5.8.8 505 root 2 0 5364K 1804K select 0:02 0.00% 0.00% nmbd 6738 root 10 0 16548K 15872K nanslp 0:01 0.00% 0.00% perl5.8.8 650 dlee 2 0 5336K 1836K select 0:01 0.00% 0.00% sshd 7778 root 10 0 16532K 15868K nanslp 0:01 0.00% 0.00% perl5.8.8 8307 root 10 0 16532K 15868K nanslp 0:01 0.00% 0.00% perl5.8.8 245 bind 2 0 2484K 1856K select 0:01 0.00% 0.00% named 241 root 2 0 1000K 668K select 0:01 0.00% 0.00% syslogd 8236 root 10 0 16516K 15856K nanslp 0:01 0.00% 0.00% perl5.8.8 10 root -2 0 0K 0K vlruwt 0:01 0.00% 0.00% vnlru last pid: 9015; load averages: 0.02, 0.02, 0.00 up 1+00:02:02 06:37:51 76 processes: 1 running, 74 sleeping, 1 zombie CPU states: 0.4% user, 0.0% nice, 0.0% system, 0.0% interrupt, 99.6% idle Mem: 95M Active, 264M Inact, 63M Wired, 21M Cache, 60M Buf, 55M Free Swap: 250M Total, 250M Free PID USERNAME PRI NICE SIZE RES STATE TIME WCPU CPU COMMAND 491 mysql 2 0 44788K 19244K poll 0:57 0.00% 0.00% mysqld 9 root 18 0 0K 0K syncer 0:16 0.00% 0.00% syncer 330 ssbdev 2 0 9744K 8188K poll 0:12 0.00% 0.00% python 248 root 2 0 1336K 868K select 0:04 0.00% 0.00% ntpd 224 root 2 0 464K 252K select 0:03 0.00% 0.00% natd 416 root 2 0 3188K 1948K select 0:03 0.00% 0.00% sendmail 6408 root 10 0 2868K 2292K nanslp 0:03 0.00% 0.00% perl5.8.8 308 root 2 0 9884K 5732K select 0:02 0.00% 0.00% httpd 6515 root 10 0 16548K 15872K nanslp 0:02 0.00% 0.00% perl5.8.8 505 root 2 0 5364K 1804K select 0:02 0.00% 0.00% nmbd 6738 root 10 0 16548K 15872K nanslp 0:01 0.00% 0.00% perl5.8.8 650 dlee 2 0 5336K 1836K select 0:01 0.00% 0.00% sshd 7778 root 10 0 16532K 15868K nanslp 0:01 0.00% 0.00% perl5.8.8 8307 root 10 0 16532K 15868K nanslp 0:01 0.00% 0.00% perl5.8.8 245 bind 2 0 2484K 1856K select 0:01 0.00% 0.00% named 241 root 2 0 1000K 668K select 0:01 0.00% 0.00% syslogd 8236 root 10 0 16516K 15856K nanslp 0:01 0.00% 0.00% perl5.8.8 10 root -2 0 0K 0K vlruwt 0:01 0.00% 0.00% vnlru last pid: 9015; load averages: 0.02, 0.02, 0.00 up 1+00:02:04 06:37:53 76 processes: 1 running, 74 sleeping, 1 zombie CPU states: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle Mem: 95M Active, 264M Inact, 63M Wired, 21M Cache, 60M Buf, 55M Free Swap: 250M Total, 250M Free PID USERNAME PRI NICE SIZE RES STATE TIME WCPU CPU COMMAND 491 mysql 2 0 44788K 19244K poll 0:57 0.00% 0.00% mysqld 9 root 18 0 0K 0K syncer 0:16 0.00% 0.00% syncer 330 ssbdev 2 0 9744K 8188K poll 0:12 0.00% 0.00% python 248 root 2 0 1336K 868K select 0:04 0.00% 0.00% ntpd 224 root 2 0 464K 252K select 0:03 0.00% 0.00% natd 416 root 2 0 3188K 1948K select 0:03 0.00% 0.00% sendmail 6408 root 10 0 2868K 2292K nanslp 0:03 0.00% 0.00% perl5.8.8 308 root 2 0 9884K 5732K select 0:02 0.00% 0.00% httpd 6515 root 10 0 16548K 15872K nanslp 0:02 0.00% 0.00% perl5.8.8 505 root 2 0 5364K 1804K select 0:02 0.00% 0.00% nmbd 6738 root 10 0 16548K 15872K nanslp 0:01 0.00% 0.00% perl5.8.8 650 dlee 2 0 5336K 1836K select 0:01 0.00% 0.00% sshd 7778 root 10 0 16532K 15868K nanslp 0:01 0.00% 0.00% perl5.8.8 8307 root 10 0 16532K 15868K nanslp 0:01 0.00% 0.00% perl5.8.8 245 bind 2 0 2484K 1856K select 0:01 0.00% 0.00% named 241 root 2 0 1000K 668K select 0:01 0.00% 0.00% syslogd 8236 root 10 0 16516K 15856K nanslp 0:01 0.00% 0.00% perl5.8.8 10 root -2 0 0K 0K vlruwt 0:01 0.00% 0.00% vnlru last pid: 9015; load averages: 0.02, 0.02, 0.00 up 1+00:02:06 06:37:55 76 processes: 1 running, 74 sleeping, 1 zombie CPU states: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle Mem: 95M Active, 264M Inact, 63M Wired, 21M Cache, 60M Buf, 55M Free Swap: 250M Total, 250M Free PID USERNAME PRI NICE SIZE RES STATE TIME WCPU CPU COMMAND 491 mysql 2 0 44788K 19244K poll 0:57 0.00% 0.00% mysqld 9 root 18 0 0K 0K syncer 0:16 0.00% 0.00% syncer 330 ssbdev 2 0 9744K 8188K poll 0:12 0.00% 0.00% python 248 root 2 0 1336K 868K select 0:04 0.00% 0.00% ntpd 224 root 2 0 464K 252K select 0:03 0.00% 0.00% natd 416 root 2 0 3188K 1948K select 0:03 0.00% 0.00% sendmail 6408 root 10 0 2868K 2292K nanslp 0:03 0.00% 0.00% perl5.8.8 308 root 2 0 9884K 5732K select 0:02 0.00% 0.00% httpd 6515 root 10 0 16548K 15872K nanslp 0:02 0.00% 0.00% perl5.8.8 505 root 2 0 5364K 1804K select 0:02 0.00% 0.00% nmbd 6738 root 10 0 16548K 15872K nanslp 0:01 0.00% 0.00% perl5.8.8 650 dlee 2 0 5336K 1836K select 0:01 0.00% 0.00% sshd 7778 root 10 0 16532K 15868K nanslp 0:01 0.00% 0.00% perl5.8.8 8307 root 10 0 16532K 15868K nanslp 0:01 0.00% 0.00% perl5.8.8 245 bind 2 0 2484K 1856K select 0:01 0.00% 0.00% named 241 root 2 0 1000K 668K select 0:01 0.00% 0.00% syslogd 8236 root 10 0 16516K 15856K nanslp 0:01 0.00% 0.00% perl5.8.8 10 root -2 0 0K 0K vlruwt 0:01 0.00% 0.00% vnlru [Boom!] -- Doug Lee dgl@dlee.org http://www.dlee.org SSB BART Group doug.lee@ssbbartgroup.com http://www.ssbbartgroup.com "Is your cucumber bitter? Throw it away. Are there briars in your path? Turn aside. That is enough. Do not go on to say, `Why were things of this sort ever brought into the world?'" --Marcus Aurelius