Skip site navigation (1)Skip section navigation (2)
Date:      Sun, 17 May 2009 07:06:57 -0400
From:      Doug Lee <dgl@dlee.org>
To:        freebsd-questions@freebsd.org
Subject:   4.11 panic every 23 hours 55 minutes or so
Message-ID:  <20090517110657.GC2706@mini.local>

next in thread | raw e-mail | index | archive | help
One of the weirder things I've seen in a while here...

OS: FreeBSD 4.11 (yeah I know, old, but generally stable)
CPU: Intel(R) Pentium(R) 4 CPU 2.00GHz
real memory  = 536608768 (524032K bytes)
Hds: IDE

Problem:  Ever since a suspitious power outage (I say suspitious
because we think a surge was also involved), this box has been
exhibiting kernel panics about every 23 hours 55 minutes, give or
take about 4 minutes either way.  Obviously hardware is suspect,
and hopefully in line for upgrade; but as FreeBSD has always proven
so stable for me, I'm curious what on earth could cause this sort
of regular panic?

It's not time of day; if I reboot at 2:00 AM, 3:55 PM, or any other
time, it's 23:55 or so later I get a panic, whenever that may be.
I think this rules out cron jobs, external attacks, and load-based
issues.

I'll provide the latest two panic logs below, followed by a few
rounds of output from `top -Sd1000', which is `top' with system
processes shown and dumb-terminal output, logged via ssh onto another
system so I could see the last report before the panic.  I have a
longer `top' log but I'll send that only on request to avoid an
even huger post.  Curiously, this is one of the few times the uptime
went past 24 hours, and also shows the current process as something
other than Idle.  I wonder if running `top' actually changed the
results.

Any info would be most welcome.  Please Cc me on responses.


----- Panic from yesterday (double panic; this is typical) -----

Fatal trap 12: page fault while in kernel mode
fault virtual address	= 0xe4f5eb02
fault code		= supervisor read, page not present
instruction pointer	= 0x8:0xc02af097
stack pointer	        = 0x10:0xc04be8d4
frame pointer	        = 0x10:0xc04be8d8
code segment		= base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, def32 1, gran 1
processor eflags	= interrupt enabled, resume, IOPL = 0
current process		= Idle
interrupt mask		= 
trap number		= 12
panic: page fault

syncing disks... 

Fatal trap 12: page fault while in kernel mode
fault virtual address	= 0x30
fault code		= supervisor read, page not present
instruction pointer	= 0x8:0xc0377070
stack pointer	        = 0x10:0xc04be6f4
frame pointer	        = 0x10:0xc04be6fc
code segment		= base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, def32 1, gran 1
processor eflags	= interrupt enabled, resume, IOPL = 0
current process		= Idle
interrupt mask		= bio 
trap number		= 12
panic: page fault
Uptime: 23h59m5s
Automatic reboot in 15 seconds - press a key on the console to abort


----- Panic from today (just one this time, NOT typical) -----

Fatal trap 12: page fault while in kernel mode
fault virtual address	= 0xe694f564
fault code		= supervisor read, page not present
instruction pointer	= 0x8:0xc02a7b73
stack pointer	        = 0x10:0xdb394da4
frame pointer	        = 0x10:0xdb394e28
code segment		= base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, def32 1, gran 1
processor eflags	= interrupt enabled, resume, IOPL = 0
current process		= 505 (nmbd)
interrupt mask		= none
trap number		= 12
panic: page fault

syncing disks... 1 
done
Uptime: 1d0h1m36s
Automatic reboot in 15 seconds - press a key on the console to abort
Rebooting...


----- top -Sd10000 from today (stopping at the above panic) -----

last pid:  9015;  load averages:  0.02,  0.02,  0.00  up 1+00:01:56    06:37:45
76 processes:  1 running, 74 sleeping, 1 zombie
CPU states:  0.0% user,  0.0% nice,  0.4% system,  0.0% interrupt, 99.6% idle
Mem: 95M Active, 264M Inact, 63M Wired, 21M Cache, 60M Buf, 55M Free
Swap: 250M Total, 250M Free

  PID USERNAME   PRI NICE  SIZE    RES STATE    TIME   WCPU    CPU COMMAND
  491 mysql        2   0 44788K 19244K poll     0:57  0.00%  0.00% mysqld
    9 root        18   0     0K     0K syncer   0:16  0.00%  0.00% syncer
  330 ssbdev       2   0  9744K  8188K poll     0:12  0.00%  0.00% python
  248 root         2   0  1336K   868K select   0:04  0.00%  0.00% ntpd
  224 root         2   0   464K   252K select   0:03  0.00%  0.00% natd
  416 root         2   0  3188K  1948K select   0:03  0.00%  0.00% sendmail
 6408 root        10   0  2868K  2292K nanslp   0:03  0.00%  0.00% perl5.8.8
  308 root         2   0  9884K  5732K select   0:02  0.00%  0.00% httpd
 6515 root        10   0 16548K 15872K nanslp   0:02  0.00%  0.00% perl5.8.8
  505 root         2   0  5364K  1796K select   0:02  0.00%  0.00% nmbd
 6738 root        10   0 16548K 15872K nanslp   0:01  0.00%  0.00% perl5.8.8
  650 dlee         2   0  5336K  1836K select   0:01  0.00%  0.00% sshd
 7778 root        10   0 16532K 15868K nanslp   0:01  0.00%  0.00% perl5.8.8
 8307 root        10   0 16532K 15868K nanslp   0:01  0.00%  0.00% perl5.8.8
  245 bind         2   0  2484K  1856K select   0:01  0.00%  0.00% named
  241 root         2   0  1000K   668K select   0:01  0.00%  0.00% syslogd
 8236 root        10   0 16516K 15856K nanslp   0:01  0.00%  0.00% perl5.8.8
   10 root        -2   0     0K     0K vlruwt   0:01  0.00%  0.00% vnlru

last pid:  9015;  load averages:  0.02,  0.02,  0.00  up 1+00:01:58    06:37:47
76 processes:  1 running, 74 sleeping, 1 zombie
CPU states:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle
Mem: 95M Active, 264M Inact, 63M Wired, 21M Cache, 60M Buf, 55M Free
Swap: 250M Total, 250M Free

  PID USERNAME   PRI NICE  SIZE    RES STATE    TIME   WCPU    CPU COMMAND
  491 mysql        2   0 44788K 19244K poll     0:57  0.00%  0.00% mysqld
    9 root        18   0     0K     0K syncer   0:16  0.00%  0.00% syncer
  330 ssbdev       2   0  9744K  8188K poll     0:12  0.00%  0.00% python
  248 root         2   0  1336K   868K select   0:04  0.00%  0.00% ntpd
  224 root         2   0   464K   252K select   0:03  0.00%  0.00% natd
  416 root         2   0  3188K  1948K select   0:03  0.00%  0.00% sendmail
 6408 root        10   0  2868K  2292K nanslp   0:03  0.00%  0.00% perl5.8.8
  308 root         2   0  9884K  5732K select   0:02  0.00%  0.00% httpd
 6515 root        10   0 16548K 15872K nanslp   0:02  0.00%  0.00% perl5.8.8
  505 root         2   0  5364K  1804K select   0:02  0.00%  0.00% nmbd
 6738 root        10   0 16548K 15872K nanslp   0:01  0.00%  0.00% perl5.8.8
  650 dlee         2   0  5336K  1836K select   0:01  0.00%  0.00% sshd
 7778 root        10   0 16532K 15868K nanslp   0:01  0.00%  0.00% perl5.8.8
 8307 root        10   0 16532K 15868K nanslp   0:01  0.00%  0.00% perl5.8.8
  245 bind         2   0  2484K  1856K select   0:01  0.00%  0.00% named
  241 root         2   0  1000K   668K select   0:01  0.00%  0.00% syslogd
 8236 root        10   0 16516K 15856K nanslp   0:01  0.00%  0.00% perl5.8.8
   10 root        -2   0     0K     0K vlruwt   0:01  0.00%  0.00% vnlru

last pid:  9015;  load averages:  0.02,  0.02,  0.00  up 1+00:02:00    06:37:49
76 processes:  1 running, 74 sleeping, 1 zombie
CPU states:  0.4% user,  0.0% nice,  0.4% system,  0.4% interrupt, 98.8% idle
Mem: 95M Active, 264M Inact, 63M Wired, 21M Cache, 60M Buf, 55M Free
Swap: 250M Total, 250M Free

  PID USERNAME   PRI NICE  SIZE    RES STATE    TIME   WCPU    CPU COMMAND
  491 mysql        2   0 44788K 19244K poll     0:57  0.00%  0.00% mysqld
    9 root        18   0     0K     0K syncer   0:16  0.00%  0.00% syncer
  330 ssbdev       2   0  9744K  8188K poll     0:12  0.00%  0.00% python
  248 root         2   0  1336K   868K select   0:04  0.00%  0.00% ntpd
  224 root         2   0   464K   252K select   0:03  0.00%  0.00% natd
  416 root         2   0  3188K  1948K select   0:03  0.00%  0.00% sendmail
 6408 root        10   0  2868K  2292K nanslp   0:03  0.00%  0.00% perl5.8.8
  308 root         2   0  9884K  5732K select   0:02  0.00%  0.00% httpd
 6515 root        10   0 16548K 15872K nanslp   0:02  0.00%  0.00% perl5.8.8
  505 root         2   0  5364K  1804K select   0:02  0.00%  0.00% nmbd
 6738 root        10   0 16548K 15872K nanslp   0:01  0.00%  0.00% perl5.8.8
  650 dlee         2   0  5336K  1836K select   0:01  0.00%  0.00% sshd
 7778 root        10   0 16532K 15868K nanslp   0:01  0.00%  0.00% perl5.8.8
 8307 root        10   0 16532K 15868K nanslp   0:01  0.00%  0.00% perl5.8.8
  245 bind         2   0  2484K  1856K select   0:01  0.00%  0.00% named
  241 root         2   0  1000K   668K select   0:01  0.00%  0.00% syslogd
 8236 root        10   0 16516K 15856K nanslp   0:01  0.00%  0.00% perl5.8.8
   10 root        -2   0     0K     0K vlruwt   0:01  0.00%  0.00% vnlru

last pid:  9015;  load averages:  0.02,  0.02,  0.00  up 1+00:02:02    06:37:51
76 processes:  1 running, 74 sleeping, 1 zombie
CPU states:  0.4% user,  0.0% nice,  0.0% system,  0.0% interrupt, 99.6% idle
Mem: 95M Active, 264M Inact, 63M Wired, 21M Cache, 60M Buf, 55M Free
Swap: 250M Total, 250M Free

  PID USERNAME   PRI NICE  SIZE    RES STATE    TIME   WCPU    CPU COMMAND
  491 mysql        2   0 44788K 19244K poll     0:57  0.00%  0.00% mysqld
    9 root        18   0     0K     0K syncer   0:16  0.00%  0.00% syncer
  330 ssbdev       2   0  9744K  8188K poll     0:12  0.00%  0.00% python
  248 root         2   0  1336K   868K select   0:04  0.00%  0.00% ntpd
  224 root         2   0   464K   252K select   0:03  0.00%  0.00% natd
  416 root         2   0  3188K  1948K select   0:03  0.00%  0.00% sendmail
 6408 root        10   0  2868K  2292K nanslp   0:03  0.00%  0.00% perl5.8.8
  308 root         2   0  9884K  5732K select   0:02  0.00%  0.00% httpd
 6515 root        10   0 16548K 15872K nanslp   0:02  0.00%  0.00% perl5.8.8
  505 root         2   0  5364K  1804K select   0:02  0.00%  0.00% nmbd
 6738 root        10   0 16548K 15872K nanslp   0:01  0.00%  0.00% perl5.8.8
  650 dlee         2   0  5336K  1836K select   0:01  0.00%  0.00% sshd
 7778 root        10   0 16532K 15868K nanslp   0:01  0.00%  0.00% perl5.8.8
 8307 root        10   0 16532K 15868K nanslp   0:01  0.00%  0.00% perl5.8.8
  245 bind         2   0  2484K  1856K select   0:01  0.00%  0.00% named
  241 root         2   0  1000K   668K select   0:01  0.00%  0.00% syslogd
 8236 root        10   0 16516K 15856K nanslp   0:01  0.00%  0.00% perl5.8.8
   10 root        -2   0     0K     0K vlruwt   0:01  0.00%  0.00% vnlru

last pid:  9015;  load averages:  0.02,  0.02,  0.00  up 1+00:02:04    06:37:53
76 processes:  1 running, 74 sleeping, 1 zombie
CPU states:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle
Mem: 95M Active, 264M Inact, 63M Wired, 21M Cache, 60M Buf, 55M Free
Swap: 250M Total, 250M Free

  PID USERNAME   PRI NICE  SIZE    RES STATE    TIME   WCPU    CPU COMMAND
  491 mysql        2   0 44788K 19244K poll     0:57  0.00%  0.00% mysqld
    9 root        18   0     0K     0K syncer   0:16  0.00%  0.00% syncer
  330 ssbdev       2   0  9744K  8188K poll     0:12  0.00%  0.00% python
  248 root         2   0  1336K   868K select   0:04  0.00%  0.00% ntpd
  224 root         2   0   464K   252K select   0:03  0.00%  0.00% natd
  416 root         2   0  3188K  1948K select   0:03  0.00%  0.00% sendmail
 6408 root        10   0  2868K  2292K nanslp   0:03  0.00%  0.00% perl5.8.8
  308 root         2   0  9884K  5732K select   0:02  0.00%  0.00% httpd
 6515 root        10   0 16548K 15872K nanslp   0:02  0.00%  0.00% perl5.8.8
  505 root         2   0  5364K  1804K select   0:02  0.00%  0.00% nmbd
 6738 root        10   0 16548K 15872K nanslp   0:01  0.00%  0.00% perl5.8.8
  650 dlee         2   0  5336K  1836K select   0:01  0.00%  0.00% sshd
 7778 root        10   0 16532K 15868K nanslp   0:01  0.00%  0.00% perl5.8.8
 8307 root        10   0 16532K 15868K nanslp   0:01  0.00%  0.00% perl5.8.8
  245 bind         2   0  2484K  1856K select   0:01  0.00%  0.00% named
  241 root         2   0  1000K   668K select   0:01  0.00%  0.00% syslogd
 8236 root        10   0 16516K 15856K nanslp   0:01  0.00%  0.00% perl5.8.8
   10 root        -2   0     0K     0K vlruwt   0:01  0.00%  0.00% vnlru

last pid:  9015;  load averages:  0.02,  0.02,  0.00  up 1+00:02:06    06:37:55
76 processes:  1 running, 74 sleeping, 1 zombie
CPU states:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle
Mem: 95M Active, 264M Inact, 63M Wired, 21M Cache, 60M Buf, 55M Free
Swap: 250M Total, 250M Free

  PID USERNAME   PRI NICE  SIZE    RES STATE    TIME   WCPU    CPU COMMAND
  491 mysql        2   0 44788K 19244K poll     0:57  0.00%  0.00% mysqld
    9 root        18   0     0K     0K syncer   0:16  0.00%  0.00% syncer
  330 ssbdev       2   0  9744K  8188K poll     0:12  0.00%  0.00% python
  248 root         2   0  1336K   868K select   0:04  0.00%  0.00% ntpd
  224 root         2   0   464K   252K select   0:03  0.00%  0.00% natd
  416 root         2   0  3188K  1948K select   0:03  0.00%  0.00% sendmail
 6408 root        10   0  2868K  2292K nanslp   0:03  0.00%  0.00% perl5.8.8
  308 root         2   0  9884K  5732K select   0:02  0.00%  0.00% httpd
 6515 root        10   0 16548K 15872K nanslp   0:02  0.00%  0.00% perl5.8.8
  505 root         2   0  5364K  1804K select   0:02  0.00%  0.00% nmbd
 6738 root        10   0 16548K 15872K nanslp   0:01  0.00%  0.00% perl5.8.8
  650 dlee         2   0  5336K  1836K select   0:01  0.00%  0.00% sshd
 7778 root        10   0 16532K 15868K nanslp   0:01  0.00%  0.00% perl5.8.8
 8307 root        10   0 16532K 15868K nanslp   0:01  0.00%  0.00% perl5.8.8
  245 bind         2   0  2484K  1856K select   0:01  0.00%  0.00% named
  241 root         2   0  1000K   668K select   0:01  0.00%  0.00% syslogd
 8236 root        10   0 16516K 15856K nanslp   0:01  0.00%  0.00% perl5.8.8
   10 root        -2   0     0K     0K vlruwt   0:01  0.00%  0.00% vnlru

[Boom!]

-- 
Doug Lee                 dgl@dlee.org                http://www.dlee.org
SSB BART Group           doug.lee@ssbbartgroup.com   http://www.ssbbartgroup.com
"Is your cucumber bitter? Throw it away. Are there briars in your
path? Turn aside. That is enough. Do not go on to say, `Why were
things of this sort ever brought into the world?'"
--Marcus Aurelius



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20090517110657.GC2706>