Date: Tue, 24 Aug 2010 10:17:16 -0400 From: Mike Tancsa <mike@sentex.net> To: freebsd-stable@freebsd.org Subject: Re: RELENG_8 panic Message-ID: <201008241417.o7OEHCwa042809@lava.sentex.ca> In-Reply-To: <201008202104.o7KL4xi8012899@lava.sentex.ca> References: <201008202104.o7KL4xi8012899@lava.sentex.ca>
next in thread | previous in thread | raw e-mail | index | archive | help
At 05:04 PM 8/20/2010, Mike Tancsa wrote:
>The box is a moderately busy LNS running mpd5. I have another box
>running the same load that has not crashed so I am wondering if its
>hardware or this box is just "lucky" ? its crashed a couple of times
>now, but the watchdog rebooted it prior to the dump being written out.
I had disabled the watchdog and was able to get a full crash dump
this time. Seems to be about 3 days between crashes
Unread portion of the kernel message buffer:
Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address = 0x24
fault code = supervisor read, page not present
instruction pointer = 0x20:0xc5d11e15
stack pointer = 0x28:0xc4e70704
frame pointer = 0x28:0xc4e70718
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, def32 1, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 0 (em1 taskq)
trap number = 12
panic: page fault
cpuid = 0
Uptime: 3d20h45m41s
Physical memory: 2036 MB
Dumping 228 MB: 212 196 180 164 148 132 116 100 84 68 52 36 20 4
Reading symbols from /boot/kernel/coretemp.ko...Reading symbols from
/boot/kernel/coretemp.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/coretemp.ko
Reading symbols from /boot/kernel/if_disc.ko...Reading symbols from
/boot/kernel/if_disc.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/if_disc.ko
Reading symbols from /boot/kernel/ipfw.ko...Reading symbols from
/boot/kernel/ipfw.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/ipfw.ko
Reading symbols from /boot/kernel/libalias.ko...Reading symbols from
/boot/kernel/libalias.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/libalias.ko
Reading symbols from /boot/kernel/ng_socket.ko...Reading symbols from
/boot/kernel/ng_socket.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/ng_socket.ko
Reading symbols from /boot/kernel/netgraph.ko...Reading symbols from
/boot/kernel/netgraph.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/netgraph.ko
Reading symbols from /boot/kernel/ng_mppc.ko...Reading symbols from
/boot/kernel/ng_mppc.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/ng_mppc.ko
Reading symbols from /boot/kernel/rc4.ko...Reading symbols from
/boot/kernel/rc4.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/rc4.ko
Reading symbols from /boot/kernel/ichwd.ko...Reading symbols from
/boot/kernel/ichwd.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/ichwd.ko
Reading symbols from /boot/kernel/ng_l2tp.ko...Reading symbols from
/boot/kernel/ng_l2tp.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/ng_l2tp.ko
Reading symbols from /boot/kernel/ng_ksocket.ko...Reading symbols
from /boot/kernel/ng_ksocket.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/ng_ksocket.ko
Reading symbols from /boot/kernel/ng_tee.ko...Reading symbols from
/boot/kernel/ng_tee.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/ng_tee.ko
Reading symbols from /boot/kernel/ng_iface.ko...Reading symbols from
/boot/kernel/ng_iface.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/ng_iface.ko
Reading symbols from /boot/kernel/ng_ppp.ko...Reading symbols from
/boot/kernel/ng_ppp.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/ng_ppp.ko
Reading symbols from /boot/kernel/ng_tcpmss.ko...Reading symbols from
/boot/kernel/ng_tcpmss.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/ng_tcpmss.ko
#0 doadump () at pcpu.h:231
231 pcpu.h: No such file or directory.
in pcpu.h
(kgdb) #0 doadump () at pcpu.h:231
#1 0xc06b0ac3 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:416
#2 0xc06b0d29 in panic (fmt=Variable "fmt" is not available.
) at /usr/src/sys/kern/kern_shutdown.c:590
#3 0xc092239c in trap_fatal (frame=0xc4e706c4, eva=36)
at /usr/src/sys/i386/i386/trap.c:938
#4 0xc0922620 in trap_pfault (frame=0xc4e706c4, usermode=0, eva=36)
at /usr/src/sys/i386/i386/trap.c:851
#5 0xc0922f0c in trap (frame=0xc4e706c4) at /usr/src/sys/i386/i386/trap.c:533
#6 0xc0904a9c in calltrap () at /usr/src/sys/i386/i386/exception.s:166
#7 0xc5d11e15 in ng_address_hook (here=0x0, item=0xc5da0540,
hook=0xc65ce400, retaddr=0)
at /usr/src/sys/modules/netgraph/netgraph/../../../netgraph/ng_base.c:3504
#8 0xc5dddbcb in ng_tee_rcvdata (hook=0xca997580, item=0xc5da0540)
at /usr/src/sys/modules/netgraph/tee/../../../netgraph/ng_tee.c:326
#9 0xc5d137c4 in ng_apply_item (node=0xc6499b80, item=0xc5da0540, rw=0)
at /usr/src/sys/modules/netgraph/netgraph/../../../netgraph/ng_base.c:2336
#10 0xc5d1279f in ng_snd_item (item=0xc5da0540, flags=Variable
"flags" is not available.
)
at /usr/src/sys/modules/netgraph/netgraph/../../../netgraph/ng_base.c:2253
#11 0xc5eca84a in ng_ppp_link_xmit (node=Variable "node" is not available.
)
at /usr/src/sys/modules/netgraph/ppp/../../../netgraph/ng_ppp.c:1350
#12 0xc5ecb605 in ng_ppp_mp_xmit (node=0xcaad1a00, item=0xc5da0540,
proto=Variable "proto" is not available.
)
at /usr/src/sys/modules/netgraph/ppp/../../../netgraph/ng_ppp.c:1998
#13 0xc5d137c4 in ng_apply_item (node=0xcaad1a00, item=0xc5da0540, rw=0)
at /usr/src/sys/modules/netgraph/netgraph/../../../netgraph/ng_base.c:2336
#14 0xc5d1279f in ng_snd_item (item=0xc5da0540, flags=Variable
"flags" is not available.
)
at /usr/src/sys/modules/netgraph/netgraph/../../../netgraph/ng_base.c:2253
#15 0xc5d137c4 in ng_apply_item (node=0xc5eeae80, item=0xc5da0540, rw=0)
at /usr/src/sys/modules/netgraph/netgraph/../../../netgraph/ng_base.c:2336
#16 0xc5d1279f in ng_snd_item (item=0xc5da0540, flags=Variable
"flags" is not available.
)
at /usr/src/sys/modules/netgraph/netgraph/../../../netgraph/ng_base.c:2253
#17 0xc5ec57e1 in ng_iface_send (ifp=0xc527d400, m=0xcb478200,
sa=Variable "sa" is not available.
)
at /usr/src/sys/modules/netgraph/iface/../../../netgraph/ng_iface.c:475
#18 0xc5ec5c08 in ng_iface_output (ifp=0xc527d400, m=0xcb478200,
dst=0xc4e70bdc, ro=0xc4e70bd4)
at /usr/src/sys/modules/netgraph/iface/../../../netgraph/ng_iface.c:410
#19 0xc077cde2 in ip_fastforward (m=0xcb478200)
at /usr/src/sys/netinet/ip_fastfwd.c:552
#20 0xc0759015 in ether_demux (ifp=0xc523c800, m=0xcb478200)
at /usr/src/sys/net/if_ethersubr.c:837
#21 0xc0759593 in ether_input (ifp=0xc523c800, m=0xcb478200)
at /usr/src/sys/net/if_ethersubr.c:760
#22 0xc0536d2a in lem_handle_rxtx (context=0xc5277000, pending=1)
at /usr/src/sys/dev/e1000/if_lem.c:3626
#23 0xc06ea2c2 in taskqueue_run (queue=0xc5266280)
at /usr/src/sys/kern/subr_taskqueue.c:239
#24 0xc06ea4cd in taskqueue_thread_loop (arg=0xc527b5e8)
at /usr/src/sys/kern/subr_taskqueue.c:360
#25 0xc0686561 in fork_exit (callout=0xc06ea410 <taskqueue_thread_loop>,
arg=0xc527b5e8, frame=0xc4e70d38) at /usr/src/sys/kern/kern_fork.c:844
#26 0xc0904b14 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:273
(kgdb)
------------------------------------------------------------------------
ps -axl
UID PID PPID CPU PRI NI VSZ RSS MWCHAN STAT TT TIME COMMAND
0 0 0 0 -68 0 0 0
- DLs ?? 13090934:27.00 [kernel]
0 1 0 0 44 0 2912 0 wait DLs ?? 5028681:45.00 [init]
0 2 0 0 -8 0 0 0
- RL ?? -14489947:-26.55 [g_event]
0 3 0 0 -8 0 0 0
- DL ?? -30170063:-16.55 [g_up]
0 4 0 0 -8 0 0 0
- DL ?? 20877214:22.00 [g_down]
0 5 0 0 -16 0 0 0
- DL ?? 19420261:44.00 [fdc0]
0 6 0 0 -60 0 0 0 waitin DL ?? 1587:45.00
[sctp_itera
0 7 0 0 -60 0 0 0 ccb_sc DL ?? 0:00.00 [xpt_thrd]
0 8 0 0 -16 0 0 0 psleep
DL ?? 7421017:48.00 [pagedaemon
0 9 0 0 -16 0 0 0 psleep DL ?? 51:45.00 [vmdaemon]
0 10 0 0 171 0 0 0
- RL ?? 25156623:14.00 [idle]
0 11 0 0 -48 0 0 0
- WL ?? -29218365:-21.55 [intr]
0 12 0 0 44 0 0 0
- RL ?? 24812477:16.00 [yarrow]
0 13 0 0 76 0 0 0 pgzero
DL ?? 103176:00.00 [pagezero]
0 14 0 0 44 0 0 0 psleep
DL ?? 31116207:18.00 [bufdaemon]
0 15 0 0 -16 0 0 0 vlruwt
DL ?? 31653040:03.00 [vnlru]
0 16 0 0 44 0 0 0
- RL ?? 7034379:11.00 [syncer]
0 17 0 0 44 0 0 0
- RL ?? -9165157:-23.55 [softdepflu
0 113 1 0 76 0 1540 0 pause Ds ?? 8264:42.00
[adjkerntz]
101 574 1 0 44 0 2948 0 select
Ds ?? -22918381:-22.55 [zebra]
101 580 1 0 44 0 3456 0 select
Ds ?? 7761146:41.00 [ospfd]
0 955 1 0 1 0 3348 0
- Rs ?? 3529521:14.00 [syslogd]
0 1153 0 0 45 0 0 0
- RL ?? 27744613:25.00 [ng_queue]
0 1160 1 0 44 0 3352 0 select Ds ?? 269548:12.00 [ntpd]
123 1161 1160 0 44 0 3352 0 select D ?? 8588592:09.00 [ntpd]
123 1163 1161 0 76 0 3352 0 select D ?? 21072:54.00 [ntpd]
0 1307 1 0 44 0 6704 0 select
Ds ?? 17624560:39.00 [sshd]
0 1315 1 0 44 0 6080 0
- Rs ?? 34559384:02.00 [sendmail]
25 1321 1 0 44 0 6080 0
pause Ds ?? 1682274:36.00 [sendmail]
0 1328 1 0 44 0 3376 0 nanslp
Ds ?? 17758332:54.00 [cron]
0 1418 1 0 76 0 3348 0 ttyin Ds+ ?? 55567:03.00 [getty]
0 1419 1 0 76 0 3348 0 ttyin Ds+ ?? 43887:36.00 [getty]
0 1420 1 0 76 0 3348 0 ttyin Ds+ ?? 50276:15.00 [getty]
0 1421 1 0 76 0 3348 0 ttyin Ds+ ?? 45877:21.00 [getty]
0 1422 1 0 76 0 3348 0 ttyin Ds+ ?? 46096:30.00 [getty]
0 1423 1 0 76 0 3348 0 ttyin Ds+ ?? 45587:15.00 [getty]
0 1424 1 0 76 0 3348 0 ttyin Ds+ ?? 44476:30.00 [getty]
0 1425 1 0 76 0 3348 0 ttyin Ds+ ?? 50589:36.00 [getty]
0 1426 1 0 44 0 3812 0 wait Ds ?? 395870:24.00 [login]
0 4087 1426 0 44 0 5652 0 pause D ?? 723658:48.00 [csh]
0 4090 4087 0 44 0 3288 0 kqread D+ ?? 64711:30.00 [tail]
0 4333 1 0 44 0 28440 0 select
Ds ?? -32284891:-49.55 [mpd5]
0 12120 1307 0 48 0 11372 0 sbwait Ds ?? 796169:42.00 [sshd]
1001 12122 12120 0 44 0 11372 0 select D ?? 1822270:12.00 [sshd]
1001 12123 12122 0 44 0 5652 0 pause Ds ?? 376237:48.00 [csh]
0 12134 12123 0 44 0 3808 0 wait D ?? 166535:15.00 [su]
0 12135 12134 0 44 0 5652 0 ttyin D+ ?? 1204515:36.00 [csh]
0 12139 1 0 45 0 34744 0
- Rs ?? -20815152:-14.55 [argus]
0 17075 1307 0 47 0 11372 0 sbwait Ds ?? 795308:24.00 [sshd]
1001 17077 17075 0 44 0 11372 0 select D ?? 369119:51.00 [sshd]
1001 17078 17077 0 44 0 5652 0 ttyin Ds+ ?? 769947:27.00 [csh]
------------------------------------------------------------------------
vmstat -s
3478376649 cpu context switches
2032445813 device interrupts
48096996 software interrupts
15353777 traps
1467200553 system calls
18 kernel threads created
14876 fork() calls
2517 vfork() calls
0 rfork() calls
0 swap pager pageins
0 swap pager pages paged in
0 swap pager pageouts
0 swap pager pages paged out
1040 vnode pager pageins
8157 vnode pager pages paged in
0 vnode pager pageouts
0 vnode pager pages paged out
0 page daemon wakeups
0 pages examined by the page daemon
237 pages reactivated
580900 copy-on-write faults
354 copy-on-write optimized faults
1024989 zero fill pages zeroed
34096 zero fill pages prezeroed
192 intransit blocking page faults
2153200 total VM faults taken
0 pages affected by kernel thread creation
2820899 pages affected by fork()
489397 pages affected by vfork()
0 pages affected by rfork()
335 pages cached
2003757 pages freed
0 pages freed by daemon
1367292 pages freed by exiting processes
10919 pages active
159341 pages inactive
65 pages in VM cache
45709 pages wired down
295745 pages free
4096 bytes per page
4892890 total name lookups
cache hits (59% pos + 7% neg) system 2% per-directory
deletions 0%, falsehits 0%, toolong 0%
------------------------------------------------------------------------
vmstat -m
Type InUse MemUse HighUse Requests Size(s)
filedesc 57 17K - 17428 16,32,256,512,1024,2048
kenv 114 8K - 117 16,32,64,128,4096
kqueue 2 2K - 10932 128,1024
CAM SIM 1 1K - 1 128
proc-args 33 2K - 87399 16,32,64,128,256
ithread 89 9K - 89 16,64,128
UART 3 2K - 3 16,256,1024
KTRACE 100 13K - 100 128
acpidev 59 2K - 59 32
linker 188 446K - 375 16,32,256,1024,2048,4096
lockf 23 2K - 857 32,64
ata_generic 1 1K - 9 16,512,1024
ip6ndp 795 75K - 47295 64,128
ip6opt 0 0K - 2 128
temp 821 247K -
20925999 16,32,64,128,256,512,1024,2048,4096
devbuf 364 1483K - 401
16,32,64,128,256,512,1024,2048,4096
module 320 21K - 320 64,128
mtx_pool 2 8K - 2 4096
ad_driver 1 1K - 1 32
ar_driver 0 0K - 6 512,2048
subproc 154 230K - 17612 256,4096
proc 2 8K - 2 4096
session 25 2K - 3169 64
pgrp 29 2K - 3386 64
cred 70 7K - 1187398 64,128
uidinfo 6 2K - 2880 64,1024
plimit 18 5K - 25406 256
sysctltmp 0 0K - 49868 16,32,64,128,4096
sysctloid 3021 94K - 3093 16,32,64,128
sysctl 0 0K - 20643 16,32,64
callout 1 256K - 1
umtx 204 13K - 204 64
p1003.1b 1 1K - 1 16
SWAP 2 413K - 2 64
bus-sc 56 90K - 1829
16,32,64,128,256,512,1024,2048,4096
bus 886 43K - 35234 16,32,64,128,1024
devstat 6 13K - 6 16,4096
eventhandler 81 4K - 81 32,64,128
kobj 214 428K - 250 2048
Per-cpu 1 1K - 1 16
rman 157 10K - 563 32,64
sbuf 0 0K - 940
16,32,64,128,256,512,1024,2048,4096
CAM periph 2 1K - 12 16,32,64,128
stack 0 0K - 2 128
taskqueue 19 1K - 19 16,64
Unitno 11 1K - 125295 16,64
iov 0 0K - 4457191 16,64,128,256
select 146 10K - 146 64
ioctlops 0 0K - 484247 16,32,64,128,256,512,1024,2048
msg 4 25K - 4 1024,4096
sem 4 6K - 4 256,512,1024,4096
shm 1 12K - 1
tty 22 11K - 34 512,2048
pts 2 1K - 12 128
mbuf_tag 1 1K - 998739350 32,64
ksem 1 4K - 1 4096
shmfd 1 4K - 1 4096
pci_link 16 2K - 16 32,128
pcb 186 84K - 161996 16,32,64,512,1024,2048,4096
soname 8 1K - 34723809 16,32,128
acl 0 0K - 32 4096
biobuf 1 2K - 45 2048
vfscache 1 512K - 1
cl_savebuf 0 0K - 3937 32,64
vfs_hash 1 256K - 1
vnodes 2 1K - 2 128
vnodemarker 0 0K - 86176 512
mount 69 3K - 132 16,32,64,128,256
BPF 403 90K - 17250 16,64,256
ether_multi 4019 121K - 157478 16,32,64
ifaddr 3233 748K - 126468 16,32,64,128,256,512,2048
ifnet 400 401K - 15820 64,128,256,512,1024,2048
clone 6 24K - 8 16,128,4096
arpcom 5 1K - 5 16
lltable 1202 300K - 47535 128,256
acpi_perf 2 1K - 2 64
acpica 1804 91K - 60858 16,32,64,128,256,512
DEVFS1 73 19K - 91 256
DEVFS3 90 12K - 109 128
routetbl 2761 247K - 117560 16,32,64,128,256,512
igmp 399 50K - 15814 128
DEVFS 10 1K - 11 16,64
DEVFSP 1 1K - 359 32
ip_moptions 2 1K - 2 32,128
in_multi 397 50K - 15741 128
in_mfilter 1 1K - 1 512
sctp_iter 0 0K - 9 256
sctp_ifn 6 1K - 6 128
sctp_ifa 11 2K - 11 128
sctp_vrf 1 1K - 1 64
sctp_a_it 0 0K - 9 16
hostcache 1 16K - 1
syncache 1 72K - 1
ip6_moptions 2 1K - 2 32,128
in6_multi 3180 422K - 125932 16,256
in6_mfilter 1 1K - 1 512
pfs_nodes 21 3K - 21 128
mld 399 50K - 15814 128
NFS FHA 1 1K - 1 1024
rpc 2 5K - 2 128,4096
savedino 0 0K - 729 256
newdirblk 0 0K - 1 32
dirrem 0 0K - 5260 32
mkdir 0 0K - 90 32
diradd 0 0K - 5442 64
freefile 0 0K - 1094 32
freeblks 0 0K - 1122 256
freefrag 0 0K - 17250 32
allocindir 1 1K - 49037 64
indirdep 2 1K - 16134 32
allocdirect 0 0K - 4060 128
bmsafemap 1 1K - 9184 64
newblk 1 1K - 53098 64,256
inodedep 1 256K - 6340 128
pagedep 1 64K - 789 64
ufs_dirhash 1146 211K - 1146 16,32,64,128,256,512
ufs_mount 12 49K - 12 256,2048,4096
UMAHash 1 2K - 4 256,512,1024,2048
GEOM 63 15K - 447 16,32,64,128,512,1024
vm_pgdata 2 65K - 2 64
acpitask 1 1K - 1 1024
atkbddev 2 1K - 2 32
kbdmux 6 10K - 6 16,256,1024,2048,4096
LED 4 1K - 4 64
apmdev 1 1K - 1 64
CAM XPT 12 2K - 26 16,32,64,1024
acpisem 15 2K - 15 64,128
isadev 21 2K - 21 64
io_apic 2 2K - 2 1024
memdesc 1 4K - 1 4096
msi 1 1K - 1 64
nexusdev 4 1K - 4 16
CAM dev queue 1 1K - 1 128
entropy 1024 64K - 1024 64
cdev 9 2K - 9 128
CAM queue 3 1K - 7 16
sigio 1 1K - 1 32
disc 1 1K - 1 16
IpFw/IpAcct 13 2K - 24 16,32,64,128
netgraph_msg 0 0K - 19647245 64,128,256,512,1024
netgraph_node 1800 225K - 188552 128
netgraph_hook 4248 531K - 401764 128
netgraph 1176 147K - 63692 64,256
netgraph_sock 80 5K - 70376 64
netgraph_path 0 0K - 10195923 16,32
netgraph_mppc 0 0K - 2 1024
netgraph_l2tp 468 101K - 35378 64,1024
netgraph_ksock 76 5K - 19333 64
netgraph_iface 392 25K - 15807 64
netgraph_ppp 392 3136K - 15807
------------------------------------------------------------------------
vmstat -z
ITEM SIZE LIMIT USED FREE REQUESTS FAILURES
UMA Kegs: 128, 0, 87, 3, 87, 0
UMA Zones: 888, 0, 87, 1, 87, 0
UMA Slabs: 284, 0, 1496, 16, 24018, 0
UMA RCntSlabs: 544, 0, 1267, 0, 1267, 0
UMA Hash: 128, 0, 2, 28, 3, 0
16 Bucket: 76, 0, 60, 40, 82, 0
32 Bucket: 140, 0, 92, 20, 114, 0
64 Bucket: 268, 0, 114, 12, 160, 14
128 Bucket: 524, 0, 244, 1, 1077, 112
VM OBJECT: 136, 0, 56609, 318, 418318, 0
MAP: 140, 0, 7, 21, 7, 0
KMAP ENTRY: 72, 57505, 25, 399, 85251, 0
MAP ENTRY: 72, 0, 930, 395, 783736, 0
DP fakepg: 72, 0, 0, 0, 0, 0
SG fakepg: 72, 0, 0, 0, 0, 0
mt_zone: 2056, 0, 249, 238, 249, 0
16: 16, 0, 6475, 630, 45472233, 0
32: 32, 0, 6482, 524,
999601063, 0
64: 64, 0, 7957, 3548, 10862360, 0
128: 128, 0, 9460, 560, 15223676, 0
256: 256, 0, 4810, 200, 376877, 0
512: 512, 0, 1238, 266, 787258, 0
1024: 1024, 0, 497, 87, 8592861, 0
2048: 2048, 0, 247, 85, 11756451, 0
4096: 4096, 0, 105, 68, 73770, 0
Files: 56, 0, 273, 330, 1297543, 0
TURNSTILE: 72, 0, 205, 35, 205, 0
umtx pi: 52, 0, 0, 0, 0, 0
PROC: 680, 0, 50, 52, 17411, 0
THREAD: 636, 0, 128, 76, 47327, 0
SLEEPQUEUE: 44, 0, 205, 90, 205, 0
VMSPACE: 232, 0, 33, 103, 17395, 0
cpuset: 40, 0, 2, 182, 2, 0
mbuf_packet: 256, 0, 2252, 189,
1770562500, 0
mbuf: 256, 0, 30660, 328,
2623840453, 0
mbuf_cluster: 2048, 25600, 2432, 14, 2432, 0
mbuf_jumbo_page: 4096, 12800, 0, 44, 266, 0
mbuf_jumbo_9k: 9216, 19200, 0, 0, 0, 0
mbuf_jumbo_16k: 16384, 12800, 0, 0, 0, 0
mbuf_ext_refcnt: 4, 0, 0, 0, 0, 0
ttyinq: 152, 0, 165, 69, 480, 0
ttyoutq: 256, 0, 88, 32, 256, 0
g_bio: 140, 0, 0, 3808, 1251337, 0
ata_request: 204, 0, 1, 1044, 301616, 0
ata_composite: 180, 0, 0, 0, 0, 0
VNODE: 268, 0, 80955, 105, 1591300, 0
VNODEPOLL: 60, 0, 1, 125, 1, 0
S VFS Cache: 72, 0, 87335, 804, 1613030, 0
L VFS Cache: 292, 0, 226, 632, 12424, 0
NAMEI: 1024, 0, 0, 48, 3452761, 0
NFSMOUNT: 524, 0, 0, 0, 0, 0
NFSNODE: 468, 0, 0, 0, 0, 0
DIRHASH: 1024, 0, 1373, 19, 1373, 0
pipe: 392, 0, 6, 44, 7635, 0
ksiginfo: 80, 0, 91, 965, 11791, 0
itimer: 220, 0, 0, 72, 97, 0
KNOTE: 72, 0, 2, 210, 10960, 0
socket: 412, 25605, 280, 89, 540850, 0
unpcb: 172, 25622, 23, 92, 12892, 0
ipq: 32, 904, 0, 0, 0, 0
udp_inpcb: 220, 25614, 84, 96, 417278, 0
udpcb: 8, 25781, 84, 322, 417278, 0
tcp_inpcb: 220, 25614, 11, 61, 1510, 0
tcpcb: 632, 25602, 11, 25, 1510, 0
tcptw: 52, 5184, 0, 216, 59, 0
syncache: 112, 15365, 0, 105, 1241, 0
hostcache: 76, 15400, 1, 149, 21, 0
tcpreass: 20, 1690, 0, 0, 0, 0
sackhole: 20, 0, 0, 169, 9, 0
sctp_ep: 860, 25600, 0, 0, 0, 0
sctp_asoc: 1484, 40000, 0, 0, 0, 0
sctp_laddr: 24, 80040, 0, 290, 10, 0
sctp_raddr: 432, 80001, 0, 0, 0, 0
sctp_chunk: 92, 400008, 0, 0, 0, 0
sctp_readq: 76, 400000, 0, 0, 0, 0
sctp_stream_msg_out: 64, 400020, 0, 0, 0, 0
sctp_asconf: 24, 400055, 0, 0, 0, 0
sctp_asconf_ack: 24, 400055, 0, 0, 0, 0
ripcb: 220, 25614, 2, 52, 32, 0
rtentry: 108, 0, 2194, 254, 79838, 0
selfd: 28, 0, 414, 475,
2819104093, 0
SWAPMETA: 276, 121576, 0, 0, 0, 0
Mountpoints: 644, 0, 5, 13, 5, 0
FFS inode: 116, 0, 80922, 159, 1591194, 0
FFS1 dinode: 128, 0, 0, 0, 0, 0
FFS2 dinode: 256, 0, 80922, 138, 1591194, 0
IPFW dynamic rule: 108, 0, 0, 0, 0, 0
NetGraph items: 36, 4130, 31, 205,
766085117, 0
NetGraph data items: 36, 531, 1, 412,
1762181728, 0
------------------------------------------------------------------------
vmstat -i
interrupt total rate
irq1: atkbd0 6 0
irq4: uart0 2642 14
irq19: atapci1 298104 1665
irq30: em0 1055249819 5895250
irq31: em1 976895242 5457515
cpu0: timer 667882096 3731184
cpu1: timer 667868364 3731108
Total 3368196273 18816738
------------------------------------------------------------------------
pstat -T
273/12328 files
0M/3071M swap space
------------------------------------------------------------------------
pstat -s
Device 512-blocks Used Avail Capacity
/dev/ad4s1b 6291200 0 6291200 0%
------------------------------------------------------------------------
iostat
iostat: kvm_read(_tk_nin): invalid address (0x0)
iostat: disabling TTY statistics
iostat: kvm_getcptime: invalid address (0x0)
iostat: disabling CPU time statistics
ad4
KB/t tps MB/s
13.27 1662 21.53
------------------------------------------------------------------------
ipcs -a
Message Queues:
T ID KEY
MODE OWNER GROUP CREATOR CGROUP CBYTES
QNUM QBYTES LSPID LRPID
STIME RTIME CTIME
Shared Memory:
T ID KEY
MODE OWNER GROUP CREATOR CGROUP NATTCH
SEGSZ CPID LPID ATIME DTIME CTIME
Semaphores:
T ID KEY
MODE OWNER GROUP CREATOR CGROUP NSEMS OTIME CTIME
------------------------------------------------------------------------
ipcs -T
msginfo:
msgmax: 16384 (max characters in a message)
msgmni: 40 (# of message queues)
msgmnb: 2048 (max characters in a message queue)
msgtql: 40 (max # of messages in system)
msgssz: 8 (size of a message segment)
msgseg: 2048 (# of message segments in system)
shminfo:
shmmax: 33554432 (max shared memory segment size)
shmmin: 1 (min shared memory segment size)
shmmni: 192 (max number of shared memory identifiers)
shmseg: 128 (max shared memory segments per process)
shmall: 8192 (max amount of shared memory in pages)
seminfo:
semmap: 30 (# of entries in semaphore map)
semmni: 10 (# of semaphore identifiers)
semmns: 60 (# of semaphores in system)
semmnu: 30 (# of undo structures in system)
semmsl: 60 (max # of semaphores per id)
semopm: 100 (max # of operations per semop call)
semume: 10 (max # of undo entries per process)
semusz: 136 (size in bytes of undo structure)
semvmx: 32767 (semaphore maximum value)
semaem: 16384 (adjust on exit max value)
------------------------------------------------------------------------
netstat -m
32912/517/33429 mbufs in use (current/cache/total)
2243/203/2446/25600 mbuf clusters in use (current/cache/total/max)
2252/189 mbuf+clusters out of packet secondary zone in use (current/cache)
0/44/44/12800 4k (page size) jumbo clusters in use (current/cache/total/max)
0/0/0/19200 9k jumbo clusters in use (current/cache/total/max)
0/0/0/12800 16k jumbo clusters in use (current/cache/total/max)
12714K/711K/13425K bytes allocated to network (current/cache/total)
0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
0/0/0 requests for jumbo clusters denied (4k/9k/16k)
0 requests for sfbufs denied
0 requests for sfbufs delayed
0 requests for I/O initiated by sendfile
0 calls to protocol drain routines
>Fatal trap 12: page fault while in kernel mode
>cpuid = 1; apic id = 01
>fault virtual address = 0x1378af
>fault code = supervisor read, page not present
>instruction pointer = 0x20:0xc5ba1190
>stack pointer = 0x28:0xe79b784c
>frame pointer = 0x28:0xe79b7864
>code segment = base 0x0, limit 0xfffff, type 0x1b
> = DPL 0, pres 1, def32 1, gran 1
>processor eflags = interrupt enabled, resume, IOPL = 0
>current process = 33060 (mpd5)
>trap number = 12
>panic: page fault
>cpuid = 1
>Uptime: 2d4h49m17s
>Physical memory: 2036 MB
>Dumping 228 MB:panic: bufwrite: buffer is not busy???
>cpuid = 1
> 213 197 181 165 149 133 117 101 85 69 53 37 21 5
>
>
>(kgdb) #0 doadump () at pcpu.h:231
>#1 0xc06b0ac3 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:416
>#2 0xc06b0d29 in panic (fmt=Variable "fmt" is not available.
>) at /usr/src/sys/kern/kern_shutdown.c:590
>#3 0xc092239c in trap_fatal (frame=0xe79b780c, eva=1276079)
> at /usr/src/sys/i386/i386/trap.c:938
>#4 0xc0922620 in trap_pfault (frame=0xe79b780c, usermode=0, eva=1276079)
> at /usr/src/sys/i386/i386/trap.c:851
>#5 0xc0922f0c in trap (frame=0xe79b780c) at /usr/src/sys/i386/i386/trap.c:533
>#6 0xc0904a9c in calltrap () at /usr/src/sys/i386/i386/exception.s:166
>#7 0xc5ba1190 in ng_ID2noderef (ID=64998)
> at /usr/src/sys/modules/netgraph/netgraph/../../../netgraph/ng_base.c:811
>#8 0xc5ba192b in ng_path2noderef (here=0xc5f3f180,
> address=0xcac2a800 "[fde6]:", destp=0xe79b7ac0, lasthook=0xe79b7abc)
> at
> /usr/src/sys/modules/netgraph/netgraph/../../../netgraph/ng_base.c:1679
>#9 0xc5ba1d40 in ng_address_path (here=0xc5f3f180, item=0xc5c597c0,
> address=0xcac2a800 "[fde6]:", retaddr=0)
> at
> /usr/src/sys/modules/netgraph/netgraph/../../../netgraph/ng_base.c:3536
>#10 0xc5b9c662 in ngc_send (so=0xc56e380c, flags=0, m=0xcaee6800,
> addr=0xca8baca0, control=0x0, td=0xc6298780)
> at /usr/src/sys/modules/netgraph/socket/../../../netgraph/ng_socket.c:296
>#11 0xc07111aa in sosend_generic (so=0xc56e380c, addr=0xca8baca0,
> uio=0xe79b7be8, top=0xcaee6800, control=0x0, flags=0, td=0xc6298780)
> at /usr/src/sys/kern/uipc_socket.c:1260
>#12 0xc070d40f in sosend (so=0xc56e380c, addr=0xca8baca0, uio=0xe79b7be8,
> top=0x0, control=0x0, flags=0, td=0xc6298780)
> at /usr/src/sys/kern/uipc_socket.c:1304
>#13 0xc0714a47 in kern_sendit (td=0xc6298780, s=24, mp=0xe79b7c5c, flags=0,
> control=0x0, segflg=UIO_USERSPACE)
> at /usr/src/sys/kern/uipc_syscalls.c:788
>#14 0xc0714c81 in sendit (td=0xc6298780, s=24, mp=0xe79b7c5c, flags=0)
> at /usr/src/sys/kern/uipc_syscalls.c:724
>#15 0xc0714d98 in sendto (td=0xc6298780, uap=0xe79b7cf8)
> at /usr/src/sys/kern/uipc_syscalls.c:840
>#16 0xc09228fa in syscall (frame=0xe79b7d38)
> at /usr/src/sys/i386/i386/trap.c:1111
>#17 0xc0904b01 in Xint0x80_syscall ()
> at /usr/src/sys/i386/i386/exception.s:264
>#18 0x00000033 in ?? ()
>Previous frame inner to this frame (corrupt stack?)
>(kgdb)
>
>
>
> ---Mike
>
>_______________________________________________
>freebsd-stable@freebsd.org mailing list
>http://lists.freebsd.org/mailman/listinfo/freebsd-stable
>To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"
--------------------------------------------------------------------
Mike Tancsa, tel +1 519 651 3400
Sentex Communications, mike@sentex.net
Providing Internet since 1994 www.sentex.net
Cambridge, Ontario Canada www.sentex.net/mike
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?201008241417.o7OEHCwa042809>