Date: Tue, 10 Dec 2013 13:09:54 +0100 From: dennis berger <db@nipsi.de> To: FreeBSD stable <freebsd-stable@freebsd.org> Subject: freebsd 9.1 kernel panic / kmem_malloc(2887680): kmem _map too small: 28092231680 total allocated Message-ID: <7E81B592-EE04-4DD6-8FD9-AE7EF1C8C31C@nipsi.de>
next in thread | raw e-mail | index | archive | help
Dear list,
in the last days our storage system crashed (kernel panic savecore: =
reboot after panic: kmem_malloc(2887680): kmem
_map too small: 28092231680 total allocated) two times. After the first =
time I've activated a crash device in order to get kernel core dump, =
which I got the second time it crashed.
Both times the crash happened at night around 3 o'clock when we clean up =
old snapshots, backup the filesystems and locatedb gets updated.
Additionally I recognized that one device name is too long and geom was =
complaining about it.=20
I=92ve renamed this to a shorter one today, but it=92s unclear if that =
was the cause. In any case I=92ve attached the error message as well.=20
Have you a hint for what the cause could be and if it=92s fixed in =
recent stable?
best,
-dennis
=3D OS =3D
* FreeBSD 9.1 release plus a patched opensolaris.ko/zfs.ko which =
includes one fix for a zfs deadlock in arc handler.
=3D Software=3D
* kernel nfsv3 with 16 kernel threads.
* istgt accessing zvols
* samba3 accessing zfs filesystems directly.
=3D Attachments =3D
=3D=3D opensolaris.ko patch mentioned above =3D=3D
--- a/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c
+++ b/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c
@@ -3720,8 +3720,16 @@ arc_lowmem(void *arg __unused, int
howto __unused)
mutex_enter(&arc_reclaim_thr_lock);
needfree =3D 1;
cv_signal(&arc_reclaim_thr_cv);
- while (needfree)
- msleep(&needfree, &arc_reclaim_thr_lock, 0, "zfs:lowmem", 0);
+
+ /*
+ * It is unsafe to block here in arbitrary threads, because
we can come
+ * here from ARC itself and may hold ARC locks and thus risk
a deadlock
+ * with ARC reclaim thread.
+ */
+ if (curproc =3D=3D pageproc) {
+ while (needfree)
+ msleep(&needfree, &arc_reclaim_thr_lock, 0, "zfs:lowmem", 0);
+ }
mutex_exit(&arc_reclaim_thr_lock);
mutex_exit(&arc_lowmem_lock);
=20
=3D=3D Geom error message ( hundreds of them ) =3D=3D=20
Dec 10 11:09:02 freenas kernel: g_dev_taste: make_dev_p() failed =
(gp->name=3Dzvol/vmpool/XXXXXXXXXXXXXXXXXX@hourly-2013-12-10_11.09.01--1ds=
2, error=3D63)
=3D=3D Kernel debugger + backtrace =3D=3D
# kgdb /boot/kernel/kernel /var/crash/vmcore.0=20
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you =
are
welcome to change it and/or distribute copies of it under certain =
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for =
details.
This GDB was configured as "amd64-marcel-freebsd"...
Unread portion of the kernel message buffer:
panic: kmem_malloc(2887680): kmem_map too small: 28092231680 total =
allocated
cpuid =3D 6
KDB: stack backtrace:
#0 0xffffffff809208a6 at kdb_backtrace+0x66
#1 0xffffffff808ea8be at panic+0x1ce
#2 0xffffffff80b44930 at vm_map_locked+0
#3 0xffffffff80b3b41a at uma_large_malloc+0x4a
#4 0xffffffff808d5a69 at malloc+0xd9
#5 0xffffffff80928931 at sbuf_extend+0x51
#6 0xffffffff809289ef at sbuf_put_byte+0x6f
#7 0xffffffff809235d3 at kvprintf+0x83
#8 0xffffffff80928b96 at sbuf_vprintf+0x26
#9 0xffffffff80928c10 at sbuf_printf+0x60
#10 0xffffffff8085cad8 at g_part_dumpconf+0x98
#11 0xffffffff80854359 at g_conf_specific+0x349
#12 0xffffffff80854e4c at g_run_events+0x1ec
#13 0xffffffff808bb9ef at fork_exit+0x11f
#14 0xffffffff80bc368e at fork_trampoline+0xe
Uptime: 4d23h40m31s
Dumping 17627 out of 49123 =
MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91%
(kgdb) bt
#0 doadump (textdump=3DVariable "textdump" is not available.
) at pcpu.h:224
#1 0xffffffff808ea3a1 in kern_reboot (howto=3D260)
at /usr/src/sys/kern/kern_shutdown.c:448
#2 0xffffffff808ea897 in panic (fmt=3D0x1 <Address 0x1 out of bounds>)
at /usr/src/sys/kern/kern_shutdown.c:636
#3 0xffffffff80b44930 in kmem_malloc (map=3DVariable "map" is not =
available.
) at /usr/src/sys/vm/vm_kern.c:304
#4 0xffffffff80b3b41a in uma_large_malloc (size=3D2887680, wait=3D2)
at /usr/src/sys/vm/uma_core.c:3067
#5 0xffffffff808d5a69 in malloc (size=3D2887680, =
mtp=3D0xffffffff811a22c0, flags=3D2)
at /usr/src/sys/kern/kern_malloc.c:492
#6 0xffffffff80928931 in sbuf_extend (s=3D0xfffffe091201d100, =
addlen=3DVariable "addlen" is not available.
)
at /usr/src/sys/kern/subr_sbuf.c:159
#7 0xffffffff809289ef in sbuf_put_byte (s=3D0xfffffe091201d100, c=3D116)
at /usr/src/sys/kern/subr_sbuf.c:369
#8 0xffffffff809235d3 in kvprintf (fmt=3D0xffffffff80e19c63 ">\n",=20
func=3D0xffffffff80928a20 <sbuf_putc_func>, arg=3D0xfffffe091201d100, =
radix=3D10,=20
ap=3DVariable "ap" is not available.
) at /usr/src/sys/kern/subr_prf.c:737
#9 0xffffffff80928b96 in sbuf_vprintf (s=3D0xfffffe091201d100, =
fmt=3DVariable "fmt" is not available.
)
at /usr/src/sys/kern/subr_sbuf.c:544
#10 0xffffffff80928c10 in sbuf_printf (s=3DVariable "s" is not =
available.
) at /usr/src/sys/kern/subr_sbuf.c:626
#11 0xffffffff8085cad8 in g_part_dumpconf (sb=3D0xfffffe091201d100,=20
indent=3D0xffffffff80e18965 "\t ", gp=3DVariable "gp" is not =
available.
)
at /usr/src/sys/geom/part/g_part.c:1993
---Type <return> to continue, or q <return> to quit---
#12 0xffffffff80854359 in g_conf_specific (sb=3D0xfffffe091201d100, =
mp=3D0x0,=20
gp=3D0x0, pp=3D0x0, cp=3D0x0) at /usr/src/sys/geom/geom_dump.c:216
#13 0xffffffff80854e4c in g_run_events () at =
/usr/src/sys/geom/geom_event.c:259
#14 0xffffffff808bb9ef in fork_exit (
callout=3D0xffffffff80856380 <g_event_procbody>, arg=3D0x0,=20
frame=3D0xffffff80003b1c40) at /usr/src/sys/kern/kern_fork.c:992
#15 0xffffffff80bc368e in fork_trampoline ()
at /usr/src/sys/amd64/amd64/exception.S:602
#16 0x0000000000000000 in ?? ()
#17 0x0000000000000000 in ?? ()
#18 0x0000000000000001 in ?? ()
#19 0x0000000000000000 in ?? ()
#20 0x0000000000000000 in ?? ()
#21 0x0000000000000000 in ?? ()
#22 0x0000000000000000 in ?? ()
#23 0x0000000000000000 in ?? ()
#24 0x0000000000000000 in ?? ()
#25 0x0000000000000000 in ?? ()
#26 0x0000000000000000 in ?? ()
#27 0x0000000000000000 in ?? ()
#28 0x0000000000000000 in ?? ()
#29 0x0000000000000000 in ?? ()
#30 0x0000000000000000 in ?? ()
---Type <return> to continue, or q <return> to quit---
#31 0x0000000000000000 in ?? ()
#32 0x0000000000000000 in ?? ()
#33 0x0000000000000000 in ?? ()
#34 0x0000000000000000 in ?? ()
#35 0x0000000000000000 in ?? ()
#36 0x0000000000000000 in ?? ()
#37 0x0000000000000000 in ?? ()
#38 0x0000000000000000 in ?? ()
#39 0x0000000000000000 in ?? ()
#40 0x000000000000000a in ?? ()
#41 0xffffffff81248000 in tdq_cpu ()
#42 0xfffffe00139c38e0 in ?? ()
#43 0x0000000000000000 in ?? ()
#44 0xffffff80003b1600 in ?? ()
#45 0xffffff80003b15a8 in ?? ()
#46 0xfffffe00139a38e0 in ?? ()
#47 0xffffffff8091352e in sched_switch (td=3D0x0, newtd=3D0x0, =
flags=3DVariable "flags" is not available.
)
at /usr/src/sys/kern/sched_ule.c:1921
Previous frame inner to this frame (corrupt stack?)
(kgdb)=20
(kgdb)=20
=3D=3D /var/log/messages =3D=3D
Dec 10 03:23:27 freenas kernel: g_dev_taste: make_dev_p() failed =
(gp->name=3Dzvol/
vmpool/XXXXXXXXXXXXXXXX@hourly-2013-12-09_12.09.03--1ds2, error=3D63)
Dec 10 03:23:27 freenas kernel: WARNING: /var/crash: GJOURNAL flag on fs =
but no=20
gjournal provider below
Dec 10 03:23:27 freenas kernel: ix3: Could not setup receive structures
Dec 10 03:23:27 freenas last message repeated 3 times
Dec 10 03:23:27 freenas savecore: reboot after panic: =
kmem_malloc(2887680): kmem
_map too small: 28092231680 total allocated
Dec 10 03:23:27 freenas savecore: writing core to vmcore.0
Dec 10 03:26:29 freenas kernel: pid 954 (fstat), uid 0, was killed: out =
of swap=20
space
=3D Information about the system right before the crash =3D
=3D=3D top -b =3D=3D
last pid: 87181; load averages: 0.28, 0.30, 0.16 up 4+23:32:13 =
03:09:00
55 processes: 1 running, 54 sleeping
Mem: 13M Active, 595M Inact, 41G Wired, 9852K Cache, 4922M Buf, 4780M =
Free
Swap: 50G Total, 50G Free
PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU =
COMMAND
1099 root 4 20 0 136M 6704K select 3 58:20 8.15% =
bacula-fd
87164 root 1 22 0 14008K 7844K zio->i 1 0:04 6.05% =
find
1009 root 32 20 0 9912K 1000K rpcsvc 14 36:28 0.00% =
nfsd
1164 root 1 20 0 88868K 10748K select 7 22:22 0.00% =
bsnmpd
1071 root 20 20 0 138M 36704K uwait 8 15:14 0.00% =
istgt
1023 root 1 20 0 14132K 500K rpcsvc 7 1:27 0.00% =
rpc.lockd
1058 root 1 20 0 51384K 1844K select 13 0:14 0.00% =
ntpd
1085 root 1 20 0 55876K 2472K select 6 0:07 0.00% =
nmbd
1109 root 1 20 0 47044K 2624K select 11 0:05 0.00% =
sendmail
1089 root 1 20 0 80728K 3692K select 14 0:02 0.00% =
winbindd
1087 root 1 20 0 83032K 3456K select 12 0:02 0.00% =
smbd
22262 root 1 20 0 91636K 7532K select 7 0:01 0.00% =
smbd
990 root 1 20 0 47040K 1696K select 10 0:01 0.00% =
rpcbind
1116 root 1 20 0 47036K 1752K nanslp 2 0:01 0.00% =
cron
1141 root 1 20 0 55824K 2484K select 0 0:01 0.00% =
sshd
942 root 1 20 0 44960K 1472K select 15 0:01 0.00% =
syslogd
1094 root 1 20 0 80652K 3732K select 0 0:01 0.00% =
winbindd
1242 root 1 20 0 80652K 3636K select 3 0:01 0.00% =
winbindd
=3D=3D vmstat -m =3D=3D =20
Type InUse MemUse HighUse Requests Size(s)
CAM dev queue 9 2K - 9 128
md_sii_data 0 0K - 4953 512
CAM XPT 523 606K - 5079 16,32,64,128,256,1024,2048
SCSI SES 1 2K - 1 2048
mps 35 459K - 125 =
16,32,64,128,256,512,1024,4096
isadev 7 1K - 7 128
UART 6 4K - 6 16,512,1024
acpiintr 1 1K - 1 64
cdev 8 2K - 8 256
acpica 6649 733K - 794997 =
16,32,64,128,256,512,1024,2048,4096
sigio 1 1K - 1 64
filedesc 89 51K - 88905 =
16,32,64,128,256,512,1024,2048,4096
kenv 82 11K - 1702 16,32,64,128
kqueue 14 16K - 12776 256,2048
proc-args 45 3K - 144142 16,32,64,128,256
hhook 2 1K - 2 128
acpitask 1 2K - 1 2048
ithread 332 53K - 332 32,128,256
CAM queue 70 44K - 6035 =
16,32,64,128,256,512,1024,2048,4096
KTRACE 100 13K - 100 128
linker 218 808K - 310 =
16,32,64,128,256,512,2048,4096
lockf 74 9K - 13384132 64,128
loginclass 3 1K - 2514 64
ip6ndp 22 2K - 27 64,128
temp 103 15K - 13035118 =
16,32,64,128,256,512,1024,2048,4096
devbuf 22477 81023K - 23254 =
16,32,64,128,256,512,1024,2048,4096
module 488 61K - 488 128
mtx_pool 2 16K - 2 =20
USBdev 42 9K - 42 64,128,512,4096
osd 35 1K - 36 16,64
USB 66 36K - 68 16,32,64,128,256,2048,4096
pmchooks 1 1K - 1 128
subproc 954 743K - 88061 512,4096
proc 2 16K - 2 =20
session 32 4K - 4047 128
pgrp 32 4K - 4190 128
cred 120 19K - 39895700 64,256
uidinfo 4 3K - 571713 128,2048
plimit 17 5K - 32899 256
sysctltmp 0 0K - 6635 16,32,64,128,256,4096
sysctloid 12804 632K - 13098 16,32,64,128
sysctl 0 0K - 727908 16,32,64
tidhash 1 16K - 1 =20
callout 15 7680K - 15 =20
umtx 6240 780K - 6240 128
p1003.1b 1 1K - 1 16
SWAP 2 7649K - 10 64
bus-sc 218 506K - 14625 =
16,32,64,128,256,512,1024,2048,4096
bus 1674 162K - 29317 =
16,32,64,128,256,512,1024,2048
devstat 1564 3153K - 1564 32,4096
eventhandler 95 8K - 95 64,128
acpisem 17 3K - 17 128
kobj 336 1344K - 35336 4096
CAM SIM 9 3K - 9 256
Per-cpu 1 1K - 1 32
CAM periph 46 12K - 562 16,32,64,128,256
rman 378 43K - 857 16,32,128
sbuf 1 1K - 384622502 =
16,32,64,128,256,512,1024,2048,4096
entropy 1024 64K - 1024 64
ctlmem 5062 10113K - 5062 128,2048
stack 0 0K - 716 256
taskqueue 251 23K - 86171 16,32,64,128,1024
Unitno 13 1K - 151273 32,64
iov 0 0K - 11645047 16,64,128,256,512
select 822 103K - 822 128
ioctlops 0 0K - 125364 =
16,32,64,128,256,512,1024,2048
msg 4 30K - 4 2048,4096
sem 4 106K - 4 2048,4096
shm 1 20K - 1 =20
tty 21 21K - 32 1024,2048
pts 0 0K - 9 256
mbuf_tag 0 0K - 12892 32,128
shmfd 1 8K - 1 =20
pcb 54 158K - 12638 16,32,128,1024,2048,4096
soname 11 2K - 328995 16,32,128
acl 0 0K - 2234889 4096
vfscache 1 8192K - 1 =20
cl_savebuf 0 0K - 141 64
export_host 6 2K - 18 256
vfs_hash 1 4096K - 1 =20
DEVFS1 4571 2286K - 5088 512
vnodes 2 1K - 2 256
DEVFS3 4605 1152K - 4697 256
mount 197 15K - 3908 16,32,64,128,256,512
vnodemarker 0 0K - 386295 512
BPF 15 2K - 15 128
ether_multi 180 10K - 216 16,32,64
ifaddr 133 34K - 133 32,64,128,256,512,4096
ifnet 16 31K - 16 128,2048
clone 6 24K - 8 16,128,4096
arpcom 6 1K - 6 16
lltable 53 21K - 387 256,512
DEVFS 22 1K - 23 16,128
DEVFSP 1 1K - 1 64
routetbl 81 10K - 10162 32,64,128,256,512
igmp 15 4K - 15 256
in_multi 7 2K - 7 256
sctp_iter 0 0K - 9 256
sctp_ifn 7 1K - 7 128
sctp_ifa 15 2K - 15 128
sctp_vrf 1 1K - 1 64
sctp_a_it 0 0K - 9 16
hostcache 1 28K - 1 =20
syncache 1 96K - 1 =20
ip6_moptions 2 1K - 2 32,256
in6_multi 67 10K - 67 32,256
in6_mfilter 1 1K - 1 1024
mld 15 2K - 15 128
NLM 1 2K - 30641 32,256,2048
rpc 98 13K - 48635539 =
16,32,64,128,256,512,1024,2048
audit_evclass 179 6K - 218 32
freework 1 1K - 1 16
newblk 1 1024K - 1 =20
bmsafemap 1 8K - 1 =20
inodedep 1 4096K - 1 =20
pagedep 1 1024K - 1 =20
ufs_dirhash 180 57K - 1271 16,32,64,128,256,512
ufs_mount 6 25K - 11 512,1024,4096
vm_pgdata 2 129K - 6 128
UMAHash 3 14K - 12 512,1024,2048,4096
memdesc 1 4K - 1 4096
NFSD srvcache 0 0K - 17222411 128
atkbddev 2 1K - 2 64
pfs_nodes 21 6K - 21 256
ctlblk 200 1600K - 200 =20
GEOM 30540 5905K - 227528 =
16,32,64,128,256,512,1024,2048,4096
ramdisk 1 4096K - 1 =20
acpidev 67 5K - 67 64
ctlpool 532 142K - 532 32,512
kbdmux 7 18K - 7 16,512,1024,2048
LED 28 2K - 28 16,128
apmdev 1 1K - 1 128
madt_table 0 0K - 1 4096
feeder 7 1K - 7 32
scsi_da 0 0K - 840 16
pci_link 16 2K - 16 64,128
acpi_perf 16 4K - 16 256
qpidrv 2 1K - 2 16
raid_data 0 0K - 29718 32,128,256
io_apic 3 6K - 3 2048
MCA 33 5K - 33 128
msi 56 7K - 56 128
nexusdev 4 1K - 4 16
md_nvidia_data 0 0K - 4953 512
ipmi 0 0K - 6 128
solaris 6052422 34083371K - 855712754 =
16,32,64,128,256,512,1024,2048,4096
kstat_data 4 1K - 4 64
=3D=3D vmstat -z =3D=3D
ITEM SIZE LIMIT USED FREE REQ FAIL SLEEP
UMA Kegs: 208, 0, 189, 15, 189, 0, 0
UMA Zones: 2432, 0, 189, 0, 189, 0, 0
UMA Slabs: 568, 0, 1028022, 1205552,67578210, 0, 0
UMA RCntSlabs: 568, 0, 64686, 1835, 263367, 0, 0
UMA Hash: 256, 0, 78, 12, 81, 0, 0
16 Bucket: 152, 0, 52, 173, 242, 0, 0
32 Bucket: 280, 0, 155, 223, 421, 1, 0
64 Bucket: 536, 0, 290, 116, 871, 137, 0
128 Bucket: 1048, 0, 5602, 2, 895393,3679513, =
0
VM OBJECT: 232, 0, 296742, 183210, 7303072, 0, 0
MAP: 232, 0, 7, 25, 7, 0, 0
KMAP ENTRY: 120, 1549473, 403200, 230967,164135153, 0, =
0
MAP ENTRY: 120, 0, 2746, 4911, 3839047, 0, 0
fakepg: 120, 0, 0, 0, 0, 0, 0
mt_zone: 4112, 0, 331, 42, 331, 0, 0
16: 16, 0, 1607034, 421230,145940609, 0, =
0
32: 32, 0, 913848, 91809,148582956, 0, =
0
64: 64, 0, 301580, 1848260,634878743, 0, =
0
128: 128, 0, 256324, 493819,132936793, 0, =
0
256: 256, 0, 114915, 126540,82219080, 0, 0
512: 512, 0, 1932913, 347085,122393860, 0, =
0
1024: 1024, 0, 2354, 7230, 6335288, 0, 0
2048: 2048, 0, 8300, 3358,36143800, 0, 0
4096: 4096, 0, 142416, 10073,14056966, 0, 0
Files: 80, 0, 353, 4102,10473433, 0, 0
TURNSTILE: 136, 0, 3121, 359, 3121, 0, 0
umtx pi: 96, 0, 0, 0, 0, 0, 0
MAC labels: 40, 0, 0, 0, 0, 0, 0
PROC: 1184, 0, 75, 801, 87183, 0, 0
THREAD: 1128, 0, 2714, 406, 46141, 0, 0
SLEEPQUEUE: 80, 0, 3121, 446, 3121, 0, 0
VMSPACE: 392, 0, 56, 2124, 87165, 0, 0
cpuset: 72, 0, 1839, 661, 45300, 0, 0
audit_record: 960, 0, 0, 0, 0, 0, 0
mbuf_packet: 256, 0, 18805, 1419,253572421, 0, =
0
mbuf: 256, 0, 49286, 3975,1345906286, 0, =
0
mbuf_cluster: 2048, 262144, 20224, 2512,19208068, 0, 0
mbuf_jumbo_page: 4096, 131072, 40, 1536,382314578, 0, =
0
mbuf_jumbo_9k: 9216, 65536, 49136, 2606,46897363, 4, 0
mbuf_jumbo_16k: 16384, 32768, 0, 0, 0, 0, 0
mbuf_ext_refcnt: 4, 0, 0, 0, 0, 0, 0
g_bio: 232, 0, 8, 7560,334229483, 0, =
0
ttyinq: 160, 0, 120, 480, 930, 0, 0
ttyoutq: 256, 0, 64, 356, 489, 0, 0
ata_request: 328, 0, 0, 0, 0, 0, 0
ata_composite: 336, 0, 0, 0, 0, 0, 0
VNODE: 480, 0, 454870, 34098,26814245, 0, 0
VNODEPOLL: 112, 0, 0, 0, 0, 0, 0
NAMEI: 1024, 0, 0, 1040,62691796, 0, 0
S VFS Cache: 108, 0, 417293, 36094,23514596, 0, 0
STS VFS Cache: 148, 0, 0, 0, 0, 0, 0
L VFS Cache: 328, 0, 69668, 26056, 3432753, 0, 0
LTS VFS Cache: 368, 0, 0, 0, 0, 0, 0
DIRHASH: 1024, 0, 169, 1655, 11763, 0, 0
NCLNODE: 568, 0, 0, 0, 0, 0, 0
Mountpoints: 792, 0, 15, 65, 16, 0, 0
pipe: 728, 0, 37, 1868, 61363, 0, 0
AIO: 208, 0, 0, 0, 0, 0, 0
AIOP: 32, 0, 0, 0, 0, 0, 0
AIOCB: 480, 0, 0, 0, 0, 0, 0
AIOL: 128, 0, 0, 0, 0, 0, 0
AIOLIO: 272, 0, 0, 0, 0, 0, 0
ksiginfo: 112, 0, 865, 2798, 140833, 0, 0
itimer: 344, 0, 1, 21, 1, 0, 0
KNOTE: 128, 0, 21, 675, 7613312, 0, 0
socket: 680, 262146, 134, 1126, 61267, 0, 0
unpcb: 240, 262144, 30, 946, 18662, 0, 0
ipq: 56, 8253, 0, 0, 0, 0, 0
udp_inpcb: 392, 262150, 63, 267, 32514, 0, 0
udpcb: 16, 262248, 63, 2121, 32514, 0, 0
tcp_inpcb: 392, 262150, 40, 520, 10074, 0, 0
tcpcb: 976, 262144, 39, 497, 10074, 0, 0
tcptw: 72, 27800, 1, 899, 1735, 0, 0
syncache: 152, 15375, 0, 550, 7380, 0, 0
hostcache: 136, 15372, 3, 557, 90, 0, 0
tcpreass: 40, 16464, 0, 1764, 2394, 0, 0
sackhole: 32, 0, 0, 2727, 92869, 0, 0
sctp_ep: 1376, 262144, 0, 0, 0, 0, 0
sctp_asoc: 2288, 40000, 0, 0, 0, 0, 0
sctp_laddr: 48, 80064, 0, 576, 14, 0, 0
sctp_raddr: 704, 80000, 0, 0, 0, 0, 0
sctp_chunk: 136, 400008, 0, 0, 0, 0, 0
sctp_readq: 104, 400032, 0, 0, 0, 0, 0
sctp_stream_msg_out: 112, 400026, 0, 0, 0, 0, 0
sctp_asconf: 40, 400008, 0, 0, 0, 0, 0
sctp_asconf_ack: 48, 400032, 0, 0, 0, 0, 0
ripcb: 392, 262150, 0, 100, 10, 0, 0
rtentry: 200, 0, 47, 200, 58, 0, 0
selfd: 56, 0, 1333, 2636,105220327, 0, =
0
SWAPMETA: 288, 116519, 0, 0, 0, 0, 0
FFS inode: 168, 0, 211949, 2397, 1336134, 0, 0
FFS1 dinode: 128, 0, 0, 0, 0, 0, 0
FFS2 dinode: 256, 0, 211949, 2356, 1336134, 0, 0
taskq_zone: 48, 0, 0, 2880, 125840, 0, 0
zio_cache: 880, 0, 10, 22154,375343426, 0, =
0
zio_link_cache: 48, 0, 5, 29731,364012142, 0, =
0
zio_buf_512: 512, 0, 0, 0, 0, 0, 0
zio_data_buf_512: 512, 0, 0, 0, 0, 0, 0
zio_buf_1024: 1024, 0, 0, 0, 0, 0, 0
zio_data_buf_1024: 1024, 0, 0, 0, 0, 0, 0
zio_buf_1536: 1536, 0, 0, 0, 0, 0, 0
zio_data_buf_1536: 1536, 0, 0, 0, 0, 0, 0
zio_buf_2048: 2048, 0, 0, 0, 0, 0, 0
zio_data_buf_2048: 2048, 0, 0, 0, 0, 0, 0
zio_buf_2560: 2560, 0, 0, 0, 0, 0, 0
zio_data_buf_2560: 2560, 0, 0, 0, 0, 0, 0
zio_buf_3072: 3072, 0, 0, 0, 0, 0, 0
zio_data_buf_3072: 3072, 0, 0, 0, 0, 0, 0
zio_buf_3584: 3584, 0, 0, 0, 0, 0, 0
zio_data_buf_3584: 3584, 0, 0, 0, 0, 0, 0
zio_buf_4096: 4096, 0, 0, 0, 0, 0, 0
zio_data_buf_4096: 4096, 0, 0, 0, 0, 0, 0
zio_buf_5120: 5120, 0, 0, 0, 0, 0, 0
zio_data_buf_5120: 5120, 0, 0, 0, 0, 0, 0
zio_buf_6144: 6144, 0, 0, 0, 0, 0, 0
zio_data_buf_6144: 6144, 0, 0, 0, 0, 0, 0
zio_buf_7168: 7168, 0, 0, 0, 0, 0, 0
zio_data_buf_7168: 7168, 0, 0, 0, 0, 0, 0
zio_buf_8192: 8192, 0, 0, 0, 0, 0, 0
zio_data_buf_8192: 8192, 0, 0, 0, 0, 0, 0
zio_buf_10240: 10240, 0, 0, 0, 0, 0, 0
zio_data_buf_10240: 10240, 0, 0, 0, 0, 0, 0
zio_buf_12288: 12288, 0, 0, 0, 0, 0, 0
zio_data_buf_12288: 12288, 0, 0, 0, 0, 0, 0
zio_buf_14336: 14336, 0, 0, 0, 0, 0, 0
zio_data_buf_14336: 14336, 0, 0, 0, 0, 0, 0
zio_buf_16384: 16384, 0, 0, 0, 0, 0, 0
zio_data_buf_16384: 16384, 0, 0, 0, 0, 0, 0
zio_buf_20480: 20480, 0, 0, 0, 0, 0, 0
zio_data_buf_20480: 20480, 0, 0, 0, 0, 0, 0
zio_buf_24576: 24576, 0, 0, 0, 0, 0, 0
zio_data_buf_24576: 24576, 0, 0, 0, 0, 0, 0
zio_buf_28672: 28672, 0, 0, 0, 0, 0, 0
zio_data_buf_28672: 28672, 0, 0, 0, 0, 0, 0
zio_buf_32768: 32768, 0, 0, 0, 0, 0, 0
zio_data_buf_32768: 32768, 0, 0, 0, 0, 0, 0
zio_buf_36864: 36864, 0, 0, 0, 0, 0, 0
zio_data_buf_36864: 36864, 0, 0, 0, 0, 0, 0
zio_buf_40960: 40960, 0, 0, 0, 0, 0, 0
zio_data_buf_40960: 40960, 0, 0, 0, 0, 0, 0
zio_buf_45056: 45056, 0, 0, 0, 0, 0, 0
zio_data_buf_45056: 45056, 0, 0, 0, 0, 0, 0
zio_buf_49152: 49152, 0, 0, 0, 0, 0, 0
zio_data_buf_49152: 49152, 0, 0, 0, 0, 0, 0
zio_buf_53248: 53248, 0, 0, 0, 0, 0, 0
zio_data_buf_53248: 53248, 0, 0, 0, 0, 0, 0
zio_buf_57344: 57344, 0, 0, 0, 0, 0, 0
zio_data_buf_57344: 57344, 0, 0, 0, 0, 0, 0
zio_buf_61440: 61440, 0, 0, 0, 0, 0, 0
zio_data_buf_61440: 61440, 0, 0, 0, 0, 0, 0
zio_buf_65536: 65536, 0, 0, 0, 0, 0, 0
zio_data_buf_65536: 65536, 0, 0, 0, 0, 0, 0
zio_buf_69632: 69632, 0, 0, 0, 0, 0, 0
zio_data_buf_69632: 69632, 0, 0, 0, 0, 0, 0
zio_buf_73728: 73728, 0, 0, 0, 0, 0, 0
zio_data_buf_73728: 73728, 0, 0, 0, 0, 0, 0
zio_buf_77824: 77824, 0, 0, 0, 0, 0, 0
zio_data_buf_77824: 77824, 0, 0, 0, 0, 0, 0
zio_buf_81920: 81920, 0, 0, 0, 0, 0, 0
zio_data_buf_81920: 81920, 0, 0, 0, 0, 0, 0
zio_buf_86016: 86016, 0, 0, 0, 0, 0, 0
zio_data_buf_86016: 86016, 0, 0, 0, 0, 0, 0
zio_buf_90112: 90112, 0, 0, 0, 0, 0, 0
zio_data_buf_90112: 90112, 0, 0, 0, 0, 0, 0
zio_buf_94208: 94208, 0, 0, 0, 0, 0, 0
zio_data_buf_94208: 94208, 0, 0, 0, 0, 0, 0
zio_buf_98304: 98304, 0, 0, 0, 0, 0, 0
zio_data_buf_98304: 98304, 0, 0, 0, 0, 0, 0
zio_buf_102400: 102400, 0, 0, 0, 0, 0, 0
zio_data_buf_102400: 102400, 0, 0, 0, 0, 0, 0
zio_buf_106496: 106496, 0, 0, 0, 0, 0, 0
zio_data_buf_106496: 106496, 0, 0, 0, 0, 0, 0
zio_buf_110592: 110592, 0, 0, 0, 0, 0, 0
zio_data_buf_110592: 110592, 0, 0, 0, 0, 0, 0
zio_buf_114688: 114688, 0, 0, 0, 0, 0, 0
zio_data_buf_114688: 114688, 0, 0, 0, 0, 0, 0
zio_buf_118784: 118784, 0, 0, 0, 0, 0, 0
zio_data_buf_118784: 118784, 0, 0, 0, 0, 0, 0
zio_buf_122880: 122880, 0, 0, 0, 0, 0, 0
zio_data_buf_122880: 122880, 0, 0, 0, 0, 0, 0
zio_buf_126976: 126976, 0, 0, 0, 0, 0, 0
zio_data_buf_126976: 126976, 0, 0, 0, 0, 0, 0
zio_buf_131072: 131072, 0, 0, 0, 0, 0, 0
zio_data_buf_131072: 131072, 0, 0, 0, 0, 0, 0
sa_cache: 80, 0, 242861, 39649,25477393, 0, 0
dnode_t: 856, 0, 1961556, 68320, 8022408, 0, 0
dmu_buf_impl_t: 224, 0, 2407116, 802807,36856756, 0, 0
arc_buf_hdr_t: 216, 0, 3108123, 1528605,58028612, 0, 0
arc_buf_t: 104, 0, 894075, 1915545,64612624, 0, 0
zil_lwb_cache: 192, 0, 3, 2657, 49153, 0, 0
zfs_znode_cache: 400, 0, 242861, 39640,25477393, 0, 0
=3D=3D /usr/local/bin/zfs-stats -a =3D=3D
------------------------------------------------------------------------
ZFS Subsystem Report Tue Dec 10 03:09:00 2013
------------------------------------------------------------------------
System Information:
Kernel Version: 901000 (osreldate)
Hardware Platform: amd64
Processor Architecture: amd64
ZFS Storage pool Version: 28
ZFS Filesystem Version: 5
FreeBSD 9.1-RELEASE #0 r243825: Tue Dec 4 09:23:10 UTC 2012 root
3:09AM up 4 days, 23:32, 0 users, load averages: 0.28, 0.30, 0.16
------------------------------------------------------------------------
System Memory:
0.03% 12.86 MiB Active, 1.25% 594.87 MiB Inact
88.67% 41.26 GiB Wired, 0.02% 9.62 MiB Cache
10.03% 4.67 GiB Free, 0.00% 352.00 KiB Gap
Real Installed: 48.00 GiB
Real Available: 99.94% 47.97 GiB
Real Managed: 97.00% 46.53 GiB
Logical Total: 48.00 GiB
Logical Used: 89.05% 42.74 GiB
Logical Free: 10.95% 5.26 GiB
Kernel Memory: 32.67 GiB
Data: 99.94% 32.65 GiB
Text: 0.06% 20.53 MiB
Kernel Memory Map: 30.99 GiB
Size: 100.00% 30.99 GiB
Free: 0.00% 568.00 KiB
------------------------------------------------------------------------
ARC Summary: (HEALTHY)
Memory Throttle Count: 0
ARC Misc:
Deleted: 53.77m
Recycle Misses: 5.11m
Mutex Misses: 554.83k
Evict Skips: 6.12m
ARC Size: 75.01% 34.15 GiB
Target Size: (Adaptive) 75.01% 34.16 GiB
Min Size (Hard Limit): 12.50% 5.69 GiB
Max Size (High Water): 8:1 45.53 GiB
ARC Size Breakdown:
Recently Used Cache Size: 61.07% 20.86 GiB
Frequently Used Cache Size: 38.93% 13.30 GiB
ARC Hash Breakdown:
Elements Max: 5.29m
Elements Current: 58.78% 3.11m
Collisions: 63.24m
Chain Max: 21
Chains: 739.52k
------------------------------------------------------------------------
ARC Efficiency: 238.62m
Cache Hit Ratio: 74.10% 176.82m
Cache Miss Ratio: 25.90% 61.80m
Actual Hit Ratio: 62.27% 148.58m
Data Demand Efficiency: 93.96% 76.52m
Data Prefetch Efficiency: 4.52% 54.66m
CACHE HITS BY CACHE LIST:
Anonymously Used: 12.32% 21.78m
Most Recently Used: 39.13% 69.19m
Most Frequently Used: 44.90% 79.39m
Most Recently Used Ghost: 0.37% 658.56k
Most Frequently Used Ghost: 3.28% 5.80m
CACHE HITS BY DATA TYPE:
Demand Data: 40.66% 71.90m
Prefetch Data: 1.40% 2.47m
Demand Metadata: 43.35% 76.65m
Prefetch Metadata: 14.59% 25.80m
CACHE MISSES BY DATA TYPE:
Demand Data: 7.47% 4.62m
Prefetch Data: 84.45% 52.19m
Demand Metadata: 6.68% 4.13m
Prefetch Metadata: 1.39% 861.23k
------------------------------------------------------------------------
L2 ARC Summary: (HEALTHY)
Passed Headroom: 19.14m
Tried Lock Failures: 3.41m
IO In Progress: 17
Low Memory Aborts: 209
Free on Write: 33.24k
Writes While Full: 39.98k
R/W Clashes: 193
Bad Checksums: 0
IO Errors: 0
SPA Mismatch: 15.80b
L2 ARC Size: (Adaptive) 30.67 GiB
Header Size: 0.17% 52.26 MiB
L2 ARC Evicts:
Lock Retries: 126
Upon Reading: 0
L2 ARC Breakdown: 61.80m
Hit Ratio: 2.96% 1.83m
Miss Ratio: 97.04% 59.97m
Feeds: 466.09k
L2 ARC Buffer:
Bytes Scanned: 321.97 TiB
Buffer Iterations: 466.09k
List Iterations: 29.11m
NULL List Iterations: 1.49m
L2 ARC Writes:
Writes Sent: 100.00% 92.12k
------------------------------------------------------------------------
File-Level Prefetch: (HEALTHY)
DMU Efficiency: 221.74m
Hit Ratio: 57.51% 127.53m
Miss Ratio: 42.49% 94.21m
Colinear: 94.21m
Hit Ratio: 0.01% 11.87k
Miss Ratio: 99.99% 94.19m
Stride: 111.20m
Hit Ratio: 99.92% 111.11m
Miss Ratio: 0.08% 90.41k
DMU Misc:
Reclaim: 94.19m
Successes: 0.64% 606.42k
Failures: 99.36% 93.59m
Streams: 16.43m
+Resets: 0.03% 4.43k
-Resets: 99.97% 16.43m
Bogus: 0
------------------------------------------------------------------------
VDEV cache is disabled
------------------------------------------------------------------------
ZFS Tunables (sysctl):
kern.maxusers 384
vm.kmem_size 49964781568
vm.kmem_size_scale 1
vm.kmem_size_min 0
vm.kmem_size_max 329853485875
vfs.zfs.l2c_only_size 24372193280
vfs.zfs.mfu_ghost_data_lsize 14966604288
vfs.zfs.mfu_ghost_metadata_lsize 2825756672
vfs.zfs.mfu_ghost_size 17792360960
vfs.zfs.mfu_data_lsize 15835021312
vfs.zfs.mfu_metadata_lsize 23130112
vfs.zfs.mfu_size 16063756288
vfs.zfs.mru_ghost_data_lsize 11579461632
vfs.zfs.mru_ghost_metadata_lsize 7082113536
vfs.zfs.mru_ghost_size 18661575168
vfs.zfs.mru_data_lsize 7865216000
vfs.zfs.mru_metadata_lsize 8403897344
vfs.zfs.mru_size 17009602560
vfs.zfs.anon_data_lsize 0
vfs.zfs.anon_metadata_lsize 0
vfs.zfs.anon_size 8093696
vfs.zfs.l2arc_norw 1
vfs.zfs.l2arc_feed_again 1
vfs.zfs.l2arc_noprefetch 1
vfs.zfs.l2arc_feed_min_ms 200
vfs.zfs.l2arc_feed_secs 1
vfs.zfs.l2arc_headroom 2
vfs.zfs.l2arc_write_boost 8388608
vfs.zfs.l2arc_write_max 8388608
vfs.zfs.arc_meta_limit 12222759936
vfs.zfs.arc_meta_used 12973058504
vfs.zfs.arc_min 6111379968
vfs.zfs.arc_max 48891039744
vfs.zfs.dedup.prefetch 1
vfs.zfs.mdcomp_disable 0
vfs.zfs.write_limit_override 0
vfs.zfs.write_limit_inflated 154529181696
vfs.zfs.write_limit_max 6438715904
vfs.zfs.write_limit_min 33554432
vfs.zfs.write_limit_shift 3
vfs.zfs.no_write_throttle 0
vfs.zfs.zfetch.array_rd_sz 1048576
vfs.zfs.zfetch.block_cap 256
vfs.zfs.zfetch.min_sec_reap 2
vfs.zfs.zfetch.max_streams 8
vfs.zfs.prefetch_disable 0
vfs.zfs.mg_alloc_failures 24
vfs.zfs.check_hostid 1
vfs.zfs.recover 0
vfs.zfs.txg.synctime_ms 1000
vfs.zfs.txg.timeout 5
vfs.zfs.vdev.cache.bshift 16
vfs.zfs.vdev.cache.size 0
vfs.zfs.vdev.cache.max 16384
vfs.zfs.vdev.write_gap_limit 4096
vfs.zfs.vdev.read_gap_limit 32768
vfs.zfs.vdev.aggregation_limit 131072
vfs.zfs.vdev.ramp_rate 2
vfs.zfs.vdev.time_shift 6
vfs.zfs.vdev.min_pending 4
vfs.zfs.vdev.max_pending 10
vfs.zfs.vdev.bio_flush_disable 0
vfs.zfs.cache_flush_disable 0
vfs.zfs.zil_replay_disable 0
vfs.zfs.zio.use_uma 0
vfs.zfs.snapshot_list_prefetch 0
vfs.zfs.version.zpl 5
vfs.zfs.version.spa 28
vfs.zfs.version.acl 1
vfs.zfs.debug 0
vfs.zfs.super_owner 0
------------------------------------------------------------------------
Dipl.-Inform. (FH)
Dennis Berger
email: db@bsdsystems.de
mobile: +491791231509
fon: +494054001817
GPG Fingerprint: 313E A4AD 72CB 2617 1BA9 7410 F954 07C0 DC3E 9578
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?7E81B592-EE04-4DD6-8FD9-AE7EF1C8C31C>