Date: Sat, 04 Jan 2014 14:50:05 +0200 From: Vladimir Sharun <atz@ukr.net> To: Current FreeBSD <freebsd-current@freebsd.org> Subject: ARC "pressured out", how to control/stabilize ? (reformatted to text/plain) Message-ID: <1388839805.123581691.q97ijp8l@frv45.ukr.net>
next in thread | raw e-mail | index | archive | help
Good day community,
We ran the system in production with r259544 on it. 128Gb RAM, 72Gb - arc_max and 42Gb - arc_min set in loader.conf.
The system ran both application (which are FS ops hungry) and 18Tb storage on this setup.
>From the start, ARC clearly shows 72Gb of memory consumed, but during next few days it pressured out to even lower than arc_min (after 14 days of uptime there's 28Gb only used by ARC. The problem is: lesser data in the ARC, lesser performance from the entire system. First two days (when ARC slowly decreases to 55-60Gb) the system run fine with excellent performance telemetry (application response time) up to 7-10 days, when it reach ~30Gb and then the performance falls off to unacceptable level.
So the question is: how to understand, who "eats" memory from ARC, and how to control this memory eating ?
top, vmstat's follow (vm.kmem_size limited to 100Gb, if not, wired will reach 107Gb approx with the same ARC pressured out)
last pid: 13273; load averages: 2.99, 1.55, 1.20 up 14+19:00:35 14:24:55
227 processes: 1 running, 226 sleeping
CPU: 4.6% user, 0.0% nice, 5.9% system, 1.0% interrupt, 88.5% idle
Mem: 667M Active, 30G Inact, 91G Wired, 1783M Cache, 3309M Buf, 933M Free
ARC: 28G Total, 2085M MFU, 20G MRU, 29M Anon, 1858M Header, 3855M Other
Swap:
$ vmstat -z
ITEM SIZE LIMIT USED FREE REQ FAIL SLEEP
UMA Kegs: 384, 0, 208, 2, 208, 0, 0
UMA Zones: 2176, 0, 208, 0, 208, 0, 0
UMA Slabs: 80, 0, 2311694, 204306,718386402, 0, 0
UMA RCntSlabs: 88, 0, 9229, 41, 18930, 0, 0
UMA Hash: 256, 0, 3, 12, 79, 0, 0
4 Bucket: 32, 0, 1785, 21090,1248973183, 0, 0
6 Bucket: 48, 0, 185, 12763,148487075, 0, 0
8 Bucket: 64, 0, 65, 20643,41333608, 0, 0
12 Bucket: 96, 0, 9665, 9482,11049020, 0, 0
16 Bucket: 128, 0, 649, 4807,11971939, 11, 0
32 Bucket: 256, 0, 12630, 50790,31362215, 33, 0
64 Bucket: 512, 0, 23203, 6781,17367241,1336666, 0
128 Bucket: 1024, 0, 95458, 48746,810293851,317422, 0
vmem btag: 56, 0, 1411085, 96316,27776938,21231, 0
VM OBJECT: 256, 0, 771497, 279598,2938160439, 0, 0
RADIX NODE: 144, 0, 2598081, 1326882,4515451399, 0, 0
MAP: 240, 0, 3, 61, 3, 0, 0
KMAP ENTRY: 128, 0, 25, 874, 25, 0, 0
MAP ENTRY: 128, 0, 25469, 45149,11668899620, 0, 0
VMSPACE: 448, 0, 235, 782,149851562, 0, 0
fakepg: 104, 0, 0, 0, 0, 0, 0
mt_zone: 4112, 0, 263, 0, 263, 0, 0
16: 16, 0, 2812047, 93027,53798617567, 0, 0
32: 32, 0, 9723566, 1887934,12748376945, 0, 0
64: 64, 0,10475477, 6543585,20857570861, 0, 0
128: 128, 0, 1848882, 490254,23475247741, 0, 0
256: 256, 0, 914734, 702251,12021626624, 0, 0
512: 512, 0, 1307, 581,10911947626, 0, 0
1024: 1024, 0, 28407, 233,234462675, 0, 0
2048: 2048, 0, 511, 2463,2135442383, 0, 0
4096: 4096, 0, 507210, 85487,572913690, 0, 0
uint64 pcpu: 8, 0, 298, 598, 298, 0, 0
SLEEPQUEUE: 80, 0, 1666, 721, 1669, 0, 0
Files: 80, 0, 4779, 10221,3345112793, 0, 0
TURNSTILE: 136, 0, 1666, 414, 1669, 0, 0
rl_entry: 40, 0, 696, 804, 696, 0, 0
umtx pi: 96, 0, 0, 0, 0, 0, 0
MAC labels: 40, 0, 0, 0, 0, 0, 0
PROC: 1208, 0, 256, 473,149840920, 0, 0
THREAD: 1168, 0, 1514, 151, 783235, 0, 0
cpuset: 72, 0, 791, 1134, 11746, 0, 0
audit_record: 1248, 0, 0, 0, 0, 0, 0
sendfile_sync: 64, 0, 0, 0, 0, 0, 0
mbuf_packet: 256, 41943045, 8248, 4127,8662647947, 0, 0
mbuf: 256, 41943045, 314, 13696,64235586097, 0, 0
mbuf_cluster: 2048, 6553600, 12375, 41, 20500, 0, 0
mbuf_jumbo_page: 4096, 3276800, 311, 2710,1565741448, 0, 0
mbuf_jumbo_9k: 9216, 970903, 0, 0, 0, 0, 0
mbuf_jumbo_16k: 16384, 546133, 0, 0, 0, 0, 0
mbuf_ext_refcnt: 4, 0, 0, 2259,166082214, 0, 0
g_bio: 248, 0, 9, 36135,5971956549, 0, 0
DMAR_MAP_ENTRY: 120, 0, 0, 0, 0, 0, 0
ttyinq: 160, 0, 240, 810, 2520, 0, 0
ttyoutq: 256, 0, 126, 729, 1311, 0, 0
FPU_save_area: 832, 0, 0, 0, 0, 0, 0
taskq_zone: 48, 0, 0, 2656,154582289, 0, 0
VNODE: 472, 0, 926451, 683325,224955145, 0, 0
VNODEPOLL: 112, 0, 0, 140, 1, 0, 0
BUF TRIE: 144, 0, 64, 105884,13961458, 0, 0
NAMEI: 1024, 0, 1, 471,9070182588, 0, 0
S VFS Cache: 108, 0, 884046, 757454,505000615, 0, 0
STS VFS Cache: 148, 0, 0, 0, 0, 0, 0
L VFS Cache: 328, 0, 5, 106747,78972025, 0, 0
LTS VFS Cache: 368, 0, 0, 0, 0, 0, 0
pipe: 744, 0, 10, 240,119813676, 0, 0
procdesc: 128, 0, 0, 0, 0, 0, 0
Mountpoints: 816, 0, 13, 67, 13, 0, 0
range_seg_cache: 64, 0, 3308624, 602274,2773749361, 0, 0
zio_cache: 920, 0, 17, 46059,27531825813, 0, 0
zio_link_cache: 48, 0, 12, 46634,21375486079, 0, 0
zio_buf_512: 512, 0, 1838128, 579528,233163218, 0, 0
zio_data_buf_512: 512, 0, 284, 468,280535356, 0, 0
zio_buf_1024: 1024, 0, 18918, 2674,70036984, 0, 0
zio_data_buf_1024: 1024, 0, 72, 244,25068218, 0, 0
zio_buf_1536: 1536, 0, 685, 111,15021333, 0, 0
zio_data_buf_1536: 1536, 0, 231, 135, 5491921, 0, 0
zio_buf_2048: 2048, 0, 464, 74,27382096, 0, 0
zio_data_buf_2048: 2048, 0, 252, 162, 3022922, 0, 0
zio_buf_2560: 2560, 0, 322, 39,16246431, 0, 0
zio_data_buf_2560: 2560, 0, 648, 44, 4092684, 0, 0
zio_buf_3072: 3072, 0, 266, 39,22464410, 0, 0
zio_data_buf_3072: 3072, 0, 272, 42, 3713205, 0, 0
zio_buf_3584: 3584, 0, 255, 12,13329596, 0, 0
zio_data_buf_3584: 3584, 0, 344, 15, 4160159, 0, 0
zio_buf_4096: 4096, 0, 11326, 31,296458917, 0, 0
zio_data_buf_4096: 4096, 0, 758, 22,55468107, 0, 0
zio_buf_5120: 5120, 0, 348, 13,14469868, 0, 0
zio_data_buf_5120: 5120, 0, 1048, 16,28617233, 0, 0
zio_buf_6144: 6144, 0, 286, 14,28793904, 0, 0
zio_data_buf_6144: 6144, 0, 1228, 12, 9882405, 0, 0
zio_buf_7168: 7168, 0, 287, 15,19650053, 0, 0
zio_data_buf_7168: 7168, 0, 551, 14, 3698354, 0, 0
zio_buf_8192: 8192, 0, 267, 15,41728685, 0, 0
zio_data_buf_8192: 8192, 0, 447, 17,13520347, 0, 0
zio_buf_10240: 10240, 0, 549, 14,25924463, 0, 0
zio_data_buf_10240: 10240, 0, 1035, 16,29565424, 0, 0
zio_buf_12288: 12288, 0, 496, 13,31221546, 0, 0
zio_data_buf_12288: 12288, 0, 1727, 14,15662238, 0, 0
zio_buf_14336: 14336, 0, 475, 11,15266392, 0, 0
zio_data_buf_14336: 14336, 0, 1000, 19,26789503, 0, 0
zio_buf_16384: 16384, 0, 774969, 25,765531478, 0, 0
zio_data_buf_16384: 16384, 0, 35652, 13,52893314, 0, 0
zio_buf_20480: 20480, 0, 890, 19,31495207, 0, 0
zio_data_buf_20480: 20480, 0, 1545, 15,36009279, 0, 0
zio_buf_24576: 24576, 0, 825, 12,36150814, 0, 0
zio_data_buf_24576: 24576, 0, 4914, 14,41401016, 0, 0
zio_buf_28672: 28672, 0, 701, 14,23285223, 0, 0
zio_data_buf_28672: 28672, 0, 1409, 21,27435301, 0, 0
zio_buf_32768: 32768, 0, 675, 13,31432326, 0, 0
zio_data_buf_32768: 32768, 0, 2415, 13,25945712, 0, 0
zio_buf_36864: 36864, 0, 622, 15,22148993, 0, 0
zio_data_buf_36864: 36864, 0, 1779, 15,25706882, 0, 0
zio_buf_40960: 40960, 0, 536, 18,23476092, 0, 0
zio_data_buf_40960: 40960, 0, 1673, 14,24666224, 0, 0
zio_buf_45056: 45056, 0, 513, 14,19162070, 0, 0
zio_data_buf_45056: 45056, 0, 1592, 16,17046396, 0, 0
zio_buf_49152: 49152, 0, 478, 17,24871123, 0, 0
zio_data_buf_49152: 49152, 0, 1419, 16,16353824, 0, 0
zio_buf_53248: 53248, 0, 467, 11,20353004, 0, 0
zio_data_buf_53248: 53248, 0, 1233, 16,17298271, 0, 0
zio_buf_57344: 57344, 0, 460, 13,23454274, 0, 0
zio_data_buf_57344: 57344, 0, 1267, 17,18596329, 0, 0
zio_buf_61440: 61440, 0, 423, 13,20273525, 0, 0
zio_data_buf_61440: 61440, 0, 1033, 15,15431244, 0, 0
zio_buf_65536: 65536, 0, 365, 13,20928232, 0, 0
zio_data_buf_65536: 65536, 0, 1040, 17,11739711, 0, 0
zio_buf_69632: 69632, 0, 351, 13,16369173, 0, 0
zio_data_buf_69632: 69632, 0, 1144, 14, 9285339, 0, 0
zio_buf_73728: 73728, 0, 344, 17,15927115, 0, 0
zio_data_buf_73728: 73728, 0, 932, 16, 6682428, 0, 0
zio_buf_77824: 77824, 0, 302, 13,13557151, 0, 0
zio_data_buf_77824: 77824, 0, 901, 15, 6842383, 0, 0
zio_buf_81920: 81920, 0, 301, 13,14443323, 0, 0
zio_data_buf_81920: 81920, 0, 909, 17, 5007810, 0, 0
zio_buf_86016: 86016, 0, 282, 11,12344084, 0, 0
zio_data_buf_86016: 86016, 0, 844, 17, 4623199, 0, 0
zio_buf_90112: 90112, 0, 306, 9,13898488, 0, 0
zio_data_buf_90112: 90112, 0, 861, 15, 6016346, 0, 0
zio_buf_94208: 94208, 0, 382, 13,13793511, 0, 0
zio_data_buf_94208: 94208, 0, 760, 18, 5061270, 0, 0
zio_buf_98304: 98304, 0, 283, 12,16091732, 0, 0
zio_data_buf_98304: 98304, 0, 752, 16, 5314097, 0, 0
zio_buf_102400: 102400, 0, 246, 13,14646577, 0, 0
zio_data_buf_102400: 102400, 0, 707, 20, 5504194, 0, 0
zio_buf_106496: 106496, 0, 237, 11,15777390, 0, 0
zio_data_buf_106496: 106496, 0, 704, 15, 4822634, 0, 0
zio_buf_110592: 110592, 0, 184, 13,15817496, 0, 0
zio_data_buf_110592: 110592, 0, 602, 18, 4925430, 0, 0
zio_buf_114688: 114688, 0, 166, 16,17045866, 0, 0
zio_data_buf_114688: 114688, 0, 532, 16, 5191669, 0, 0
zio_buf_118784: 118784, 0, 170, 15,12659006, 0, 0
zio_data_buf_118784: 118784, 0, 481, 17, 2124105, 0, 0
zio_buf_122880: 122880, 0, 196, 15,12948958, 0, 0
zio_data_buf_122880: 122880, 0, 524, 20, 2727770, 0, 0
zio_buf_126976: 126976, 0, 177, 12,17364224, 0, 0
zio_data_buf_126976: 126976, 0, 516, 13, 1926732, 0, 0
zio_buf_131072: 131072, 0, 179, 14,340379531, 0, 0
zio_data_buf_131072: 131072, 0, 488217, 9,287155442, 0, 0
lz4_ctx: 16384, 0, 0, 64,459560153, 0, 0
sa_cache: 80, 0, 925296, 694854,221400673, 0, 0
dnode_t: 864, 0, 3286406, 492342,164927567, 0, 0
dmu_buf_impl_t: 224, 0, 2735768, 1283814,1130573685, 0, 0
arc_buf_hdr_t: 216, 0, 8241154, 594326,869545121, 0, 0
arc_buf_t: 72, 0, 930962, 479568,1220508333, 0, 0
zil_lwb_cache: 192, 0, 30, 5830,45306321, 0, 0
zfs_znode_cache: 368, 0, 925295, 690235,221400672, 0, 0
ksiginfo: 112, 0, 720, 2535, 2627344, 0, 0
itimer: 352, 0, 0, 473, 4261, 0, 0
pf states: 296, 500006, 112, 473, 2146938, 0, 0
pf state keys: 88, 0, 118, 2042, 2220665, 0, 0
pf source nodes: 136, 500018, 0, 0, 0, 0, 0
pf mtags: 40, 0, 0, 0, 0, 0, 0
pf table entries: 160, 200000, 16, 59, 16, 0, 0
pf table counters: 64, 0, 0, 0, 0, 0, 0
pf frags: 80, 0, 0, 0, 0, 0, 0
pf frag entries: 32, 40000, 0, 0, 0, 0, 0
pf state scrubs: 40, 0, 0, 0, 0, 0, 0
KNOTE: 128, 0, 3586, 7636,4249425530, 0, 0
socket: 696, 4192805, 10988, 14362,617302776, 0, 0
unpcb: 240, 4192816, 397, 1491, 3964687, 0, 0
ipq: 56, 204835, 0, 0, 0, 0, 0
udp_inpcb: 392, 4192810, 161, 659,33932308, 0, 0
udpcb: 16, 4192955, 161, 2098,33932308, 0, 0
tcp_inpcb: 392, 4192810, 20662, 18868,579405676, 0, 0
tcpcb: 1024, 4192808, 4067, 7453,579405676, 0, 0
tcptw: 88, 27810, 16595, 11215,259638269,257834306, 0
syncache: 160, 15375, 0, 800,369362350, 0, 0
hostcache: 136, 15370, 54, 932, 2020, 0, 0
tcpreass: 40, 409600, 0, 2000, 121958, 0, 0
sackhole: 32, 0, 0, 1375, 25994, 0, 0
sctp_ep: 1400, 4192806, 0, 100, 103, 0, 0
sctp_asoc: 2344, 40000, 0, 24, 103, 0, 0
sctp_laddr: 48, 80012, 0, 415, 3, 0, 0
sctp_raddr: 720, 80000, 0, 130, 103, 0, 0
sctp_chunk: 136, 400026, 0, 435, 11, 0, 0
sctp_readq: 104, 400026, 0, 0, 0, 0, 0
sctp_stream_msg_out: 104, 400026, 0, 1406, 103, 0, 0
sctp_asconf: 40, 400000, 0, 0, 0, 0, 0
sctp_asconf_ack: 48, 400060, 0, 0, 0, 0, 0
ripcb: 392, 4192810, 0, 0, 0, 0, 0
rtentry: 200, 0, 8, 312, 8, 0, 0
selfd: 56, 0, 1288, 2120,9888371057, 0, 0
SWAPMETA: 288, 16337503, 0, 0, 0, 0, 0
FFS inode: 168, 0, 1028, 1180, 3550806, 0, 0
FFS1 dinode: 128, 0, 0, 0, 0, 0, 0
FFS2 dinode: 256, 0, 1028, 1177, 3550806, 0, 0
cyclic_id_cache: 64, 0, 0, 558, 8, 0, 0
dtrace_state_cache: 4096, 0, 0, 5, 8, 0, 0
taskq_zone: 48, 0, 0, 0, 0, 0, 0
NCLNODE: 528, 0, 0, 0, 0, 0, 0
$ vmstat -m
Type InUse MemUse HighUse Requests Size(s)
devbuf 17816 38148K - 18364 16,32,64,128,256,512,1024,2048,4096
temp 83 13K - 363465851 16,32,64,128,256,512,1024,2048,4096
acpiintr 1 1K - 1 64
module 196 25K - 196 128
mtx_pool 2 16K - 2
acpica 175642 18132K - 482200 16,32,64,128,256,512,1024,2048,4096
CAM periph 88 22K - 142 16,32,64,128,256
osd 28 1K - 2691953 16,32,64
CAM queue 61 632K - 239 16,32,64,128,256,512,1024
pmchooks 1 1K - 1 128
acpitask 1 8K - 1
acpisem 26 4K - 26 128
kbdmux 7 18K - 7 16,512,1024,2048
pgrp 36 5K - 496601 128
session 32 4K - 33701 128
proc 2 512K - 2
subproc 989 1401K - 149847035 512,4096
cred 124 20K - 4925850262 64,256
plimit 18 5K - 33674 256
uidinfo 8 65K - 140875 128
LED 28 2K - 28 16,128
md_disk 1 2K - 1 2048
raid_data 0 0K - 1584 32,128,256
mps 13 1603K - 585860 16,32,64,128,256,512,1024,2048,4096
USB 37 47K - 43 16,32,128,256,512,1024,2048,4096
USBdev 31 8K - 31 32,64,128,512,4096
sysctl 0 0K - 29680596 16,32,64
sysctloid 7556 374K - 7704 16,32,64,128
sysctltmp 0 0K - 31532343 16,32,64,128,256,2048
tidhash 1 512K - 1
callout 13 4232K - 13
umtx 3330 417K - 3336 128
p1003.1b 1 1K - 1 16
bus 950 123K - 20210 16,32,64,128,256,1024,2048
bus-sc 129 241K - 4112 16,32,64,128,256,512,1024,2048,4096
devstat 114 230K - 114 32,4096
eventhandler 98 8K - 98 64,128
CAM dev queue 9 1K - 9 64
acpidev 162 11K - 162 64
kobj 92 368K - 2386 4096
Per-cpu 1 1K - 1 32
CAM SIM 9 3K - 9 256
rman 248 28K - 662 32,128
sbuf 1 1K - 62594 16,32,64,128,256,512,1024,2048,4096
CAM XPT 388 35K - 6374 16,32,64,128,256,512,1024,2048
CAM DEV 52 104K - 97 2048
stack 0 0K - 40 256
taskqueue 259 41K - 347 16,32,64,128,256
Unitno 25 2K - 521437312 32,64
vmem 3 1184K - 16 256,512,1024,2048,4096
ioctlops 0 0K - 2301578270 16,32,64,128,256,512,1024,2048,4096
select 687 86K - 687 128
iov 0 0K - 3070408744 16,32,64,128,256,512,2048
msg 4 30K - 4 2048,4096
sem 4 106K - 4 2048,4096
shm 1 20K - 1
tty 23 23K - 93 1024,4096
pts 3 1K - 33 256
mbuf_tag 0 0K - 2 32
shmfd 1 8K - 1
soname 5 1K - 44996938125 16,32,128
pcb 1508 16579K - 216776778 16,32,128,256,1024,2048
md_nvidia_data 0 0K - 264 512
acl 0 0K - 2073 4096
vfscache 1 32768K - 1
cl_savebuf 0 0K - 2176560 64
vfs_hash 1 16384K - 1
vnodes 1 1K - 6 32,256
mount 150 6K - 540 16,32,64,128,256,1024
CAM CCB 5 10K - 1978405860 2048
vnodemarker 0 0K - 860570 512
BPF 4 1K - 4 128
md_sii_data 0 0K - 264 512
ifnet 5 9K - 5 128,2048
ifaddr 30 7K - 30 32,256,4096
ether_multi 11 1K - 12 16,64
clone 3 1K - 3 128
arpcom 3 1K - 3 16
lltable 76 20K - 14079 256,512
entropy 1026 65K - 9778336 32,64,4096
scsi_da 0 0K - 6483 32,64,512
routetbl 55 11K - 53761245 32,128,256,512
igmp 4 1K - 4 256
in_multi 2 1K - 2 256
UART 9 7K - 9 16,1024
CAM path 95 3K - 997 32
SCSI ENC 297 388K - 5602301 16,64,256,1024,2048,4096
sctp_map 0 0K - 206 16
sctp_stro 0 0K - 103 1024
sctp_a_it 0 0K - 3 16
sctp_atcl 0 0K - 206 512
sctp_atky 0 0K - 309 32,64
sctp_athm 0 0K - 206 16
sctp_vrf 1 1K - 1 64
sctp_ifa 4 1K - 4 128
sctp_ifn 2 1K - 2 128
sctp_timw 31 8K - 31 256
sctp_iter 0 0K - 3 256
hostcache 1 28K - 1
syncache 1 64K - 1
pf_hash 3 2880K - 3
pf_ifnet 8 4K - 38 256,2048
pf_rule 32 27K - 32 128,1024
pf_osfp 1184 122K - 1184 64,128
pf_table 5 10K - 10 2048
audit_evclass 187 6K - 228 32
vm_pgdata 1 16384K - 1
UMAHash 76 19947K - 497 512,1024,2048,4096
pci_link 16 2K - 16 64,128
acpi_perf 12 6K - 12 512
DEVFS2 302 5K - 332 16
DEVFS3 659 165K - 857 256
DEVFS1 302 151K - 463 512
isadev 7 1K - 7 128
memdesc 1 4K - 1 4096
DEVFS_RULE 1 1K - 1 64
DEVFS 47 1K - 50 16,128
DEVFSP 1 1K - 18 64
cdev 7 2K - 7 256
pfs_nodes 21 6K - 21 256
filedesc 701689 15308K - 151391230 16,32,64,128,256,512,1024,2048,4096
sigio 5 1K - 5 64
filecaps 0 0K - 376693052 16,64
kdtrace 1774 395K - 150625287 64,256
kenv 96 12K - 112 16,32,64,128
apmdev 1 1K - 1 128
madt_table 0 0K - 1 4096
kqueue 14 110K - 7001897 256,512,2048,4096
proc-args 237 23K - 1744195324 16,32,64,128,256
pfs_vncache 0 0K - 17 64
hhook 2 1K - 2 256
ithread 156 28K - 156 32,128,256
GEOM 1310 250K - 15089 16,32,64,128,256,512,1024,2048
io_apic 2 4K - 2 2048
prison 2 5K - 2 16,4096
MCA 31 6K - 31 128,256
KTRACE 100 13K - 100 128
msi 16 2K - 16 128
nexusdev 3 1K - 3 16
linker 404 885K - 587 16,32,64,128,256,512,1024,2048,4096
lockf 41 6K - 3660069993 64,128,256,512
loginclass 3 1K - 22891 64
rpc 1 1K - 1 256
kstat_data 6 1K - 6 64
solaris 25382156 3763478K - 72008554115 16,32,64,128,256,512,1024,2048,4096
ufs_mount 3 13K - 3 512,4096
nullfs_hash 1 131072K - 1
nullfs_node 1 1K - 1 64
nullfs_mount 1 1K - 1 32
cyclic 48 4K - 50 16,64,128
fbt 29000 3881K - 29000 128
SDT 26 2K - 26 16,64
$ zfs-stats -a
------------------------------------------------------------------------
ZFS Subsystem Report Sat Jan 4 14:27:19 2014
------------------------------------------------------------------------
System Information:
Kernel Version: 1100004 (osreldate)
Hardware Platform: amd64
Processor Architecture: amd64
ZFS Storage pool Version: 5000
ZFS Filesystem Version: 5
FreeBSD 11.0-CURRENT #5 r259544: Fri Dec 20 11:13:21 EET 2013 root
14:27 up 14 days, 19:03, 2 users, load averages: 1,27 1,56 1,27
------------------------------------------------------------------------
System Memory:
0.54% 693.36 MiB Active, 24.32% 30.31 GiB Inact
72.98% 90.96 GiB Wired, 1.42% 1.78 GiB Cache
0.70% 898.79 MiB Free, 0.04% 48.04 MiB Gap
Real Installed: 128.00 GiB
Real Available: 99.96% 127.95 GiB
Real Managed: 97.41% 124.65 GiB
Logical Total: 128.00 GiB
Logical Used: 74.25% 95.04 GiB
Logical Free: 25.75% 32.96 GiB
Kernel Memory: 3.92 GiB
Data: 99.65% 3.90 GiB
Text: 0.35% 14.00 MiB
Kernel Memory Map: 100.00 GiB
Size: 75.00% 75.00 GiB
Free: 25.00% 25.00 GiB
------------------------------------------------------------------------
ARC Summary: (HEALTHY)
Memory Throttle Count: 0
ARC Misc:
Deleted: 425.50m
Recycle Misses: 234.05m
Mutex Misses: 3.60m
Evict Skips: 502.81m
ARC Size: 38.91% 28.02 GiB
Target Size: (Adaptive) 58.33% 42.00 GiB
Min Size (Hard Limit): 58.33% 42.00 GiB
Max Size (High Water): 1:1 72.00 GiB
ARC Size Breakdown:
Recently Used Cache Size: 48.81% 20.50 GiB
Frequently Used Cache Size: 51.19% 21.50 GiB
ARC Hash Breakdown:
Elements Max: 8.83m
Elements Current: 93.29% 8.24m
Collisions: 1.27b
Chain Max: 27
Chains: 1.76m
------------------------------------------------------------------------
ARC Efficiency: 146.52b
Cache Hit Ratio: 99.62% 145.96b
Cache Miss Ratio: 0.38% 556.49m
Actual Hit Ratio: 99.52% 145.81b
Data Demand Efficiency: 92.35% 4.49b
Data Prefetch Efficiency: 8.56% 64.91m
CACHE HITS BY CACHE LIST:
Most Recently Used: 0.98% 1.43b
Most Frequently Used: 98.92% 144.38b
Most Recently Used Ghost: 0.01% 20.56m
Most Frequently Used Ghost: 0.10% 152.70m
CACHE HITS BY DATA TYPE:
Demand Data: 2.84% 4.15b
Prefetch Data: 0.00% 5.56m
Demand Metadata: 97.06% 141.67b
Prefetch Metadata: 0.10% 143.60m
CACHE MISSES BY DATA TYPE:
Demand Data: 61.74% 343.57m
Prefetch Data: 10.67% 59.35m
Demand Metadata: 25.82% 143.71m
Prefetch Metadata: 1.77% 9.86m
------------------------------------------------------------------------
L2 ARC Summary: (HEALTHY)
Passed Headroom: 27.09m
Tried Lock Failures: 916.86m
IO In Progress: 13.17k
Low Memory Aborts: 301.48k
Free on Write: 1.31m
Writes While Full: 226.67k
R/W Clashes: 99.81k
Bad Checksums: 0
IO Errors: 0
SPA Mismatch: 2.32b
L2 ARC Size: (Adaptive) 278.93 GiB
Header Size: 0.26% 750.98 MiB
L2 ARC Evicts:
Lock Retries: 6.89k
Upon Reading: 2
L2 ARC Breakdown: 556.49m
Hit Ratio: 39.35% 218.97m
Miss Ratio: 60.65% 337.52m
Feeds: 969.70k
L2 ARC Buffer:
Bytes Scanned: 4.23 PiB
Buffer Iterations: 969.70k
List Iterations: 60.83m
NULL List Iterations: 1.63m
L2 ARC Writes:
Writes Sent: 100.00% 965.91k
------------------------------------------------------------------------
File-Level Prefetch: (HEALTHY)
DMU Efficiency: 171.97b
Hit Ratio: 68.01% 116.95b
Miss Ratio: 31.99% 55.02b
Colinear: 55.02b
Hit Ratio: 0.00% 111.39k
Miss Ratio: 100.00% 55.02b
Stride: 116.77b
Hit Ratio: 99.95% 116.72b
Miss Ratio: 0.05% 56.07m
DMU Misc:
Reclaim: 55.02b
Successes: 0.11% 60.47m
Failures: 99.89% 54.96b
Streams: 199.20m
+Resets: 0.87% 1.73m
-Resets: 99.13% 197.47m
Bogus: 0
------------------------------------------------------------------------
VDEV cache is disabled
------------------------------------------------------------------------
ZFS Tunables (sysctl):
kern.maxusers 8525
vm.kmem_size 107374182400
vm.kmem_size_scale 1
vm.kmem_size_min 0
vm.kmem_size_max 1319413950874
vfs.zfs.arc_max 77309411328
vfs.zfs.arc_min 45097156608
vfs.zfs.arc_meta_used 19327359728
vfs.zfs.arc_meta_limit 19327352832
vfs.zfs.l2arc_write_max 33554432
vfs.zfs.l2arc_write_boost 134217728
vfs.zfs.l2arc_headroom 2
vfs.zfs.l2arc_feed_secs 1
vfs.zfs.l2arc_feed_min_ms 200
vfs.zfs.l2arc_noprefetch 1
vfs.zfs.l2arc_feed_again 1
vfs.zfs.l2arc_norw 1
vfs.zfs.anon_size 47123968
vfs.zfs.anon_metadata_lsize 0
vfs.zfs.anon_data_lsize 0
vfs.zfs.mru_size 21964981248
vfs.zfs.mru_metadata_lsize 4259141120
vfs.zfs.mru_data_lsize 10711522816
vfs.zfs.mru_ghost_size 23117245952
vfs.zfs.mru_ghost_metadata_lsize 23086417920
vfs.zfs.mru_ghost_data_lsize 30828032
vfs.zfs.mfu_size 2190997504
vfs.zfs.mfu_metadata_lsize 409456128
vfs.zfs.mfu_data_lsize 949248
vfs.zfs.mfu_ghost_size 21976749568
vfs.zfs.mfu_ghost_metadata_lsize 21912697344
vfs.zfs.mfu_ghost_data_lsize 64052224
vfs.zfs.l2c_only_size 292763954176
vfs.zfs.dedup.prefetch 1
vfs.zfs.nopwrite_enabled 1
vfs.zfs.mdcomp_disable 0
vfs.zfs.prefetch_disable 0
vfs.zfs.zfetch.max_streams 8
vfs.zfs.zfetch.min_sec_reap 2
vfs.zfs.zfetch.block_cap 256
vfs.zfs.zfetch.array_rd_sz 1048576
vfs.zfs.top_maxinflight 32
vfs.zfs.resilver_delay 2
vfs.zfs.scrub_delay 4
vfs.zfs.scan_idle 50
vfs.zfs.scan_min_time_ms 1000
vfs.zfs.free_min_time_ms 1000
vfs.zfs.resilver_min_time_ms 3000
vfs.zfs.no_scrub_io 0
vfs.zfs.no_scrub_prefetch 0
vfs.zfs.metaslab.gang_bang 131073
vfs.zfs.metaslab.debug_load 0
vfs.zfs.metaslab.debug_unload 0
vfs.zfs.metaslab.df_alloc_threshold 131072
vfs.zfs.metaslab.df_free_pct 4
vfs.zfs.metaslab.min_alloc_size 10485760
vfs.zfs.metaslab.load_pct 50
vfs.zfs.metaslab.unload_delay 8
vfs.zfs.metaslab.preload_limit 3
vfs.zfs.metaslab.preload_enabled 1
vfs.zfs.metaslab.weight_factor_enable 0
vfs.zfs.condense_pct 200
vfs.zfs.mg_alloc_failures 18
vfs.zfs.mg_noalloc_threshold 0
vfs.zfs.write_to_degraded 0
vfs.zfs.check_hostid 1
vfs.zfs.recover 0
vfs.zfs.deadman_synctime_ms 1000000
vfs.zfs.deadman_checktime_ms 5000
vfs.zfs.deadman_enabled 1
vfs.zfs.spa_asize_inflation 24
vfs.zfs.txg.timeout 10
vfs.zfs.vdev.cache.max 16384
vfs.zfs.vdev.cache.size 0
vfs.zfs.vdev.cache.bshift 16
vfs.zfs.vdev.trim_on_init 1
vfs.zfs.vdev.mirror.rotating_inc 0
vfs.zfs.vdev.mirror.rotating_seek_inc 5
vfs.zfs.vdev.mirror.rotating_seek_offset1048576
vfs.zfs.vdev.mirror.non_rotating_inc 0
vfs.zfs.vdev.mirror.non_rotating_seek_inc1
vfs.zfs.vdev.max_active 1000
vfs.zfs.vdev.sync_read_min_active 10
vfs.zfs.vdev.sync_read_max_active 10
vfs.zfs.vdev.sync_write_min_active 10
vfs.zfs.vdev.sync_write_max_active 10
vfs.zfs.vdev.async_read_min_active 1
vfs.zfs.vdev.async_read_max_active 3
vfs.zfs.vdev.async_write_min_active 1
vfs.zfs.vdev.async_write_max_active 10
vfs.zfs.vdev.scrub_min_active 1
vfs.zfs.vdev.scrub_max_active 2
vfs.zfs.vdev.aggregation_limit 131072
vfs.zfs.vdev.read_gap_limit 32768
vfs.zfs.vdev.write_gap_limit 4096
vfs.zfs.vdev.bio_flush_disable 0
vfs.zfs.vdev.bio_delete_disable 0
vfs.zfs.vdev.trim_max_bytes 2147483648
vfs.zfs.vdev.trim_max_pending 64
vfs.zfs.max_auto_ashift 13
vfs.zfs.zil_replay_disable 0
vfs.zfs.cache_flush_disable 0
vfs.zfs.zio.use_uma 1
vfs.zfs.zio.exclude_metadata 0
vfs.zfs.sync_pass_deferred_free 2
vfs.zfs.sync_pass_dont_compress 5
vfs.zfs.sync_pass_rewrite 2
vfs.zfs.snapshot_list_prefetch 0
vfs.zfs.super_owner 0
vfs.zfs.debug 0
vfs.zfs.version.ioctl 3
vfs.zfs.version.acl 1
vfs.zfs.version.spa 5000
vfs.zfs.version.zpl 5
vfs.zfs.trim.enabled 1
vfs.zfs.trim.txg_delay 32
vfs.zfs.trim.timeout 30
vfs.zfs.trim.max_interval 1
------------------------------------------------------------------------
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1388839805.123581691.q97ijp8l>