Skip site navigation (1)Skip section navigation (2)
Date:      Mon, 2 Jan 2012 04:59:29 -0800
From:      Jeremy Chadwick <freebsd@jdc.parodius.com>
To:        Victor Balada Diaz <victor@bsdes.net>
Cc:        freebsd-stable@freebsd.org
Subject:   Re: Performance problems with pagedaemon
Message-ID:  <20120102125929.GA89966@icarus.home.lan>
In-Reply-To: <20120102122549.GB17398@equilibrium.bsdes.net>
References:  <20120102122549.GB17398@equilibrium.bsdes.net>

next in thread | previous in thread | raw e-mail | index | archive | help
On Mon, Jan 02, 2012 at 01:25:49PM +0100, Victor Balada Diaz wrote:
> Hello,
> 
> First: Happy new year to everybody!
> 
> Last year i had a problem with pagedaemon that i reported here[1] but i had
> no replies. The problem went away and now it's back to beat me again.
> 
> Seems that mysql creates some bad interactions with pagedaemon and system starts
> to get slow for disk access. Eg, accessing a man page takes a few seconds. If
> i stop mysql and start it again, it recovers again. As this is a problem that have
> appeared just twice (on October and now), it's not easy to reproduce and test.
> 
> Disk I/O as per gstat is not high, not many context switches, etc. Bottleneck
> seems to be CPU. I've reported more information on october's mail[1].
> 
> What can i do to debug it? Seems that maybe the most interesting thing would be
> enabling lock profiling and wait for the system to fail again, but is there anything
> that i can do/report that would be helpful to track the problem down now that's
> failing?
> 
> Thanks a lot.
> Regards.
> 
> [1]: http://lists.freebsd.org/pipermail/freebsd-stable/2011-October/064248.html

The only things I can think of are output from the following:

* vmstat -z
* sysctl -a vm
* top  (mainly for mysqld)
* top -H -S

Is this machine in question still running 8.1-RELEASE?  If so, have you
tried upgrading it to 8.2-STABLE?  I believe there have been some VM
adjustments, but I'd have to go back and examine commits.

Taken from our mysql server.  Note that the mysql storage filesystem is
stored on ZFS as well, which has added benefits/complexities yet we see
no issue.

FreeBSD XXXXXXXXXXXXXXXXX 8.2-STABLE FreeBSD 8.2-STABLE #0: Tue Dec 13 00:00:16 PST 2011 root@XXXXXXXXXXXXXXXXX:/usr/obj/usr/src/sys/X7SBI_RELENG_8_amd64  amd64

real memory  = 8589934592 (8192 MB)
avail memory = 8250040320 (7867 MB)

Mem: 539M Active, 2559M Inact, 3218M Wired, 4K Cache, 828M Buf, 1603M Free
Swap: 16G Total, 16G Free
  PID USERNAME  THR PRI NICE   SIZE    RES STATE   C   TIME   WCPU COMMAND
 1940 mysql      16  20    0   781M   401M sigwai  1 158:40  0.00% [mysqld]

top -H -S shows no indication pagedaemon is "going crazy".

loader.conf tunings:

# Increase maximum allocatable memory on a process to 2.5GB.
# We don't choose 8GB (our amount of RAM) since that would
# exhaust all memory, and result in a kernel panic.  Maximum
# stack size is set to 256MB.  One can view these settings
# using limits(1).
#
# dfldsiz = Initial data size limit (bytes)
# maxdsiz = Maximum data size limit (bytes)
# dflssiz = Initial stack size limit (bytes)
# maxssiz = Maximum stack size limit (bytes)
#
kern.maxdsiz="2560M"
kern.dfldsiz="2560M"
kern.maxssiz="256M"

# Limit ZFS ARC maximum.
# NOTE #1: In 8.2-RELEASE and onward, vm.kmem_size_scale defaults to 1,
# which means vm.kmem_size should match the amount of RAM installed
# in the system.  If using an earlier FreeBSD release, be sure to set
# vm.kmem_size manually to the amount of RAM you have.
# NOTE #2: Do not set vm.kmem_size to 2x that of physical RAM, otherwise
# vfs.zfs.arc_max effectively becomes halved.
# http://lists.freebsd.org/pipermail/freebsd-fs/2011-March/010875.html
vfs.zfs.arc_max="5120M"


my.cnf tunings:

[mysqld]
; http://dev.mysql.com/doc/refman/5.5/en/table-cache.html
table_open_cache        = 2048
max_connections         = 512

tmp_table_size          = 64M
max_allowed_packet      = 32M
key_buffer_size         = 64M
join_buffer_size        = 16M
sort_buffer_size        = 16M
read_buffer_size        = 16M
query_cache_size        = 64M
query_cache_limit       = 32M

innodb_file_per_table
innodb_buffer_pool_size         = 512M
innodb_log_file_size            = 128M
innodb_log_buffer_size          = 8M
innodb_additional_mem_pool_size = 20M


sysctl -a vm output:

vm.vmtotal:
System wide totals computed every five seconds: (values in kilobytes)
===============================================
Processes:              (RUNQ: 1 Disk Wait: 1 Page Wait: 0 Sleep: 59)
Virtual Memory:         (Total: 1075323828K Active: 1483408K)
Real Memory:            (Total: 2970948K Active: 585264K)
Shared Virtual Memory:  (Total: 17884K Active: 6756K)
Shared Real Memory:     (Total: 7500K Active: 6564K)
Free Memory Pages:      1641708K

vm.loadavg: { 0.00 0.00 0.00 }
vm.v_free_min: 12824
vm.v_free_target: 53985
vm.v_free_reserved: 2689
vm.v_inactive_target: 80977
vm.v_cache_min: 53985
vm.v_cache_max: 107970
vm.v_pageout_free_min: 34
vm.pageout_algorithm: 0
vm.swap_enabled: 1
vm.md_malloc_wait: 0
vm.kmem_map_free: 2897080320
vm.kmem_map_size: 2349465600
vm.kmem_size_scale: 1
vm.kmem_size_max: 329853485875
vm.kmem_size_min: 0
vm.kmem_size: 8303726592
vm.nswapdev: 1
vm.dmmax: 32
vm.swap_async_max: 4
vm.overcommit: 0
vm.swap_reserved: 11426652160
vm.swap_total: 17179869184
vm.zone_count: 195
vm.swap_idle_threshold2: 10
vm.swap_idle_threshold1: 2
vm.kstacks: 475
vm.kstack_cache_size: 128
vm.exec_map_entries: 16
vm.stats.misc.zero_page_count: 158
vm.stats.misc.cnt_prezero: 0
vm.stats.vm.v_kthreadpages: 0
vm.stats.vm.v_rforkpages: 0
vm.stats.vm.v_vforkpages: 35572524
vm.stats.vm.v_forkpages: 131458827
vm.stats.vm.v_kthreads: 22
vm.stats.vm.v_rforks: 0
vm.stats.vm.v_vforks: 159338
vm.stats.vm.v_forks: 711839
vm.stats.vm.v_interrupt_free_min: 2
vm.stats.vm.v_pageout_free_min: 34
vm.stats.vm.v_cache_max: 107970
vm.stats.vm.v_cache_min: 53985
vm.stats.vm.v_cache_count: 1
vm.stats.vm.v_inactive_count: 655146
vm.stats.vm.v_inactive_target: 80977
vm.stats.vm.v_active_count: 137837
vm.stats.vm.v_wire_count: 823816
vm.stats.vm.v_free_count: 410409
vm.stats.vm.v_free_min: 12824
vm.stats.vm.v_free_target: 53985
vm.stats.vm.v_free_reserved: 2689
vm.stats.vm.v_page_count: 2027277
vm.stats.vm.v_page_size: 4096
vm.stats.vm.v_tfree: 488602276
vm.stats.vm.v_pfree: 167595871
vm.stats.vm.v_dfree: 0
vm.stats.vm.v_tcached: 42510
vm.stats.vm.v_pdpages: 0
vm.stats.vm.v_pdwakeups: 0
vm.stats.vm.v_reactivated: 8878
vm.stats.vm.v_intrans: 19805
vm.stats.vm.v_vnodepgsout: 1
vm.stats.vm.v_vnodepgsin: 58873
vm.stats.vm.v_vnodeout: 1
vm.stats.vm.v_vnodein: 47194
vm.stats.vm.v_swappgsout: 0
vm.stats.vm.v_swappgsin: 0
vm.stats.vm.v_swapout: 0
vm.stats.vm.v_swapin: 0
vm.stats.vm.v_ozfod: 511
vm.stats.vm.v_zfod: 231124379
vm.stats.vm.v_cow_optim: 41799
vm.stats.vm.v_cow_faults: 39666798
vm.stats.vm.v_vm_faults: 299438319
vm.stats.sys.v_soft: 411033343
vm.stats.sys.v_intr: 263204059
vm.stats.sys.v_syscall: 3214862746
vm.stats.sys.v_trap: 645468188
vm.stats.sys.v_swtch: 2209989736
vm.stats.object.bypasses: 524363
vm.stats.object.collapses: 3306408
vm.v_free_severe: 7756
vm.old_msync: 0
vm.msync_flush_flags: 0
vm.boot_pages: 48
vm.max_wired: 666530
vm.pageout_lock_miss: 0
vm.disable_swapspace_pageouts: 0
vm.defer_swapspace_pageouts: 0
vm.swap_idle_enabled: 0
vm.pageout_stats_interval: 5
vm.pageout_full_stats_interval: 20
vm.pageout_stats_max: 53985
vm.max_launder: 32
vm.phys_segs:
SEGMENT 0:

start:     0x1000
end:       0x97000
free list: 0xffffffff80718da8

SEGMENT 1:

start:     0x8ad000
end:       0x1000000
free list: 0xffffffff80718da8

SEGMENT 2:

start:     0x1000000
end:       0xcfe70000
free list: 0xffffffff80718a00

SEGMENT 3:

start:     0x100000000
end:       0x21f8b4000
free list: 0xffffffff80718a00

vm.phys_free:
FREE LIST 0:

  ORDER (SIZE)  |  NUMBER
                |  POOL 0  |  POOL 1  |  POOL 2
--            -- --      -- --      -- --      --
  12 ( 16384K)  |       0  |       0  |       0
  11 (  8192K)  |       0  |       0  |       0
  10 (  4096K)  |      22  |       0  |       0
   9 (  2048K)  |      74  |       0  |       0
   8 (  1024K)  |      10  |       0  |       0
   7 (   512K)  |      15  |       1  |       0
   6 (   256K)  |      10  |       1  |       0
   5 (   128K)  |       4  |       1  |       0
   4 (    64K)  |       1  |       0  |       0
   3 (    32K)  |       1  |       0  |       0
   2 (    16K)  |       0  |       0  |       0
   1 (     8K)  |       0  |       1  |       0
   0 (     4K)  |       0  |       1  |       0

FREE LIST 1:

  ORDER (SIZE)  |  NUMBER
                |  POOL 0  |  POOL 1  |  POOL 2
--            -- --      -- --      -- --      --
  12 ( 16384K)  |       0  |       0  |       0
  11 (  8192K)  |       0  |       0  |       0
  10 (  4096K)  |       0  |       0  |       0
   9 (  2048K)  |       0  |       0  |       0
   8 (  1024K)  |       0  |       0  |       0
   7 (   512K)  |       0  |       0  |       0
   6 (   256K)  |       0  |       0  |       0
   5 (   128K)  |       0  |       0  |       0
   4 (    64K)  |       0  |       0  |       0
   3 (    32K)  |       0  |       0  |       0
   2 (    16K)  |       1  |       0  |       0
   1 (     8K)  |       0  |       0  |       0
   0 (     4K)  |       0  |       0  |       0

vm.reserv.reclaimed: 246
vm.reserv.partpopq:
LEVEL     SIZE  NUMBER

   -1: 1377964K,   1226

vm.reserv.freed: 2823528
vm.reserv.broken: 5973
vm.idlezero_enable: 0
vm.kvm_free: 539964207104
vm.kvm_size: 549755809792
vm.pmap.pmap_collect_active: 0
vm.pmap.pmap_collect_inactive: 0
vm.pmap.pv_entry_spare: 5609
vm.pmap.pv_entry_allocs: 932709470
vm.pmap.pv_entry_frees: 932651743
vm.pmap.pc_chunk_tryfail: 0
vm.pmap.pc_chunk_frees: 5562480
vm.pmap.pc_chunk_allocs: 5562857
vm.pmap.pc_chunk_count: 377
vm.pmap.pv_entry_count: 57727
vm.pmap.pdpe.demotions: 0
vm.pmap.pde.promotions: 3234381
vm.pmap.pde.p_failures: 36313949
vm.pmap.pde.mappings: 10185
vm.pmap.pde.demotions: 3202630
vm.pmap.shpgperproc: 200
vm.pmap.pv_entry_max: 3260077
vm.pmap.pg_ps_enabled: 1
vm.pmap.pat_works: 1


vmstat -z output:

ITEM                     SIZE     LIMIT      USED      FREE  REQUESTS  FAILURES

UMA Kegs:                 208,        0,      193,       11,      193,        0
UMA Zones:                320,        0,      193,       11,      193,        0
UMA Slabs:                568,        0,    91344,      573, 22063422,        0
UMA RCntSlabs:            568,        0,     1652,        0,     1652,        0
UMA Hash:                 256,        0,       79,       11,       81,        0
16 Bucket:                152,        0,      198,        2,      198,        0
32 Bucket:                280,        0,      220,        4,      220,        0
64 Bucket:                536,        0,      237,        1,      237,      324
128 Bucket:              1048,        0,     5090,        1,     5090,     3812
VM OBJECT:                216,        0,   112042,     1682, 19218174,        0
MAP:                      232,        0,        7,       25,        7,        0
KMAP ENTRY:               120,   278070,     2358,     8802, 56999587,        0
MAP ENTRY:                120,        0,     1254,     2063, 33749125,        0
DP fakepg:                120,        0,        0,        0,        0,        0
SG fakepg:                120,        0,        0,        0,        0,        0
mt_zone:                 2056,        0,      203,       34,      203,        0
16:                        16,        0,     5939,     4309, 812209016,        0
32:                        32,        0,     2778,     9948, 36156302,        0
64:                        64,        0,    15479,   124465, 559617065,        0
128:                      128,        0,     5998,     6124, 500883081,        0
256:                      256,        0,     1542,   105408, 116395928,        0
512:                      512,        0,     5900,     5195, 30945149,        0
1024:                    1024,        0,      203,     1789, 10984378,        0
2048:                    2048,        0,      449,     4339, 14614660,        0
4096:                    4096,        0,      789,      784, 11198444,        0
Files:                     80,        0,      780,      795, 19941086,        0
TURNSTILE:                136,        0,      778,      102,      778,        0
umtx pi:                   96,        0,        0,        0,        0,        0
MAC labels:                40,        0,        0,        0,        0,        0
PROC:                    1136,        0,       62,      181,   871274,        0
THREAD:                  1096,        0,      475,      302,  1185892,        0
SLEEPQUEUE:                80,        0,      778,      150,      778,        0
VMSPACE:                  392,        0,       41,      189,   871256,        0
cpuset:                    72,        0,        2,       98,        2,        0
audit_record:             952,        0,        0,        0,        0,        0
mbuf_packet:              256,        0,     2046,      770, 119097913,        0
mbuf:                     256,        0,        2,      902, 596890710,        0
mbuf_cluster:            2048,    25600,     2816,      116,     2834,        0
mbuf_jumbo_page:         4096,    12800,        0,      186,  8204717,        0
mbuf_jumbo_9k:           9216,     6400,        0,        0,        0,        0
mbuf_jumbo_16k:         16384,     3200,        0,        0,        0,        0
mbuf_ext_refcnt:            4,        0,        0,        0,        0,        0
g_bio:                    232,        0,        0,    12624, 132751934,        0
ttyinq:                   160,        0,      225,      207,     1260,        0
ttyoutq:                  256,        0,      117,      153,      663,        0
ata_request:              320,        0,        0,       84,      458,        0
ata_composite:            336,        0,        0,        0,        0,        0
VNODE:                    472,        0,   162386,      758,  7441920,        0
VNODEPOLL:                112,        0,        0,        0,        0,        0
NAMEI:                   1024,        0,        0,       96, 56473208,        0
S VFS Cache:              108,        0,   178642,      647,  8950044,        0
L VFS Cache:              328,        0,      860,     2428,   128661,        0
DIRHASH:                 1024,        0,      930,      134,     1315,        0
NFSMOUNT:                 632,        0,        1,       11,        1,        0
NFSNODE:                  688,        0,        6,       19,        6,        0
pipe:                     728,        0,       40,      115,   292578,        0
ksiginfo:                 112,        0,      232,      824,    86359,        0
itimer:                   344,        0,        1,       54,        6,        0
KNOTE:                    128,        0,      137,      414, 13517474,        0
socket:                   680,    25602,      114,      180,  1932153,        0
unpcb:                    240,    25600,       49,      207,   121492,        0
ipq:                       56,      819,        0,        0,        0,        0
udp_inpcb:                336,    25608,       19,      168,   351143,        0
udpcb:                     16,    25704,       19,      821,   351143,        0
tcp_inpcb:                336,    25608,       49,      633,  1226932,        0
tcpcb:                    944,    25600,       36,      124,  1226932,        0
tcptw:                     72,     5150,       13,      837,   479105,        0
syncache:                 144,    15366,        0,      156,  1191373,        0
hostcache:                136,    15372,        2,      166,      267,        0
tcpreass:                  40,     1680,        0,      336,      703,        0
sackhole:                  32,        0,        0,      505,      725,        0
ripcb:                    336,    25608,        8,      146,   232583,        0
rtentry:                  200,        0,       12,       64,       12,        0
selfd:                     56,        0,      257,      751, 629664344,        0
SWAPMETA:                 288,   116519,        0,        0,        0,        0
ip4flow:                   56,    99351,        0,        0,        0,        0
ip6flow:                   80,    99360,        0,        0,        0,        0
Mountpoints:              752,        0,        9,       16,        9,        0
FFS inode:                168,        0,   157347,     4925,  7264635,        0
FFS1 dinode:              128,        0,        0,        0,        0,        0
FFS2 dinode:              256,        0,   157347,     4848,  7264635,        0
taskq_zone:                48,        0,        0,     1008,   392677,        0
zio_cache:                880,        0,        1,    23083, 693486370,        0
zio_link_cache:            48,        0,        0,    23472, 267139845,        0
zio_buf_512:              512,        0,        0,        0,        0,        0
zio_data_buf_512:         512,        0,        0,        0,        0,        0
zio_buf_1024:            1024,        0,        0,        0,        0,        0
zio_data_buf_1024:       1024,        0,        0,        0,        0,        0
zio_buf_1536:            1536,        0,        0,        0,        0,        0
zio_data_buf_1536:       1536,        0,        0,        0,        0,        0
zio_buf_2048:            2048,        0,        0,        0,        0,        0
zio_data_buf_2048:       2048,        0,        0,        0,        0,        0
zio_buf_2560:            2560,        0,        0,        0,        0,        0
zio_data_buf_2560:       2560,        0,        0,        0,        0,        0
zio_buf_3072:            3072,        0,        0,        0,        0,        0
zio_data_buf_3072:       3072,        0,        0,        0,        0,        0
zio_buf_3584:            3584,        0,        0,        0,        0,        0
zio_data_buf_3584:       3584,        0,        0,        0,        0,        0
zio_buf_4096:            4096,        0,        0,        0,        0,        0
zio_data_buf_4096:       4096,        0,        0,        0,        0,        0
zio_buf_5120:            5120,        0,        0,        0,        0,        0
zio_data_buf_5120:       5120,        0,        0,        0,        0,        0
zio_buf_6144:            6144,        0,        0,        0,        0,        0
zio_data_buf_6144:       6144,        0,        0,        0,        0,        0
zio_buf_7168:            7168,        0,        0,        0,        0,        0
zio_data_buf_7168:       7168,        0,        0,        0,        0,        0
zio_buf_8192:            8192,        0,        0,        0,        0,        0
zio_data_buf_8192:       8192,        0,        0,        0,        0,        0
zio_buf_10240:          10240,        0,        0,        0,        0,        0
zio_data_buf_10240:     10240,        0,        0,        0,        0,        0
zio_buf_12288:          12288,        0,        0,        0,        0,        0
zio_data_buf_12288:     12288,        0,        0,        0,        0,        0
zio_buf_14336:          14336,        0,        0,        0,        0,        0
zio_data_buf_14336:     14336,        0,        0,        0,        0,        0
zio_buf_16384:          16384,        0,        0,        0,        0,        0
zio_data_buf_16384:     16384,        0,        0,        0,        0,        0
zio_buf_20480:          20480,        0,        0,        0,        0,        0
zio_data_buf_20480:     20480,        0,        0,        0,        0,        0
zio_buf_24576:          24576,        0,        0,        0,        0,        0
zio_data_buf_24576:     24576,        0,        0,        0,        0,        0
zio_buf_28672:          28672,        0,        0,        0,        0,        0
zio_data_buf_28672:     28672,        0,        0,        0,        0,        0
zio_buf_32768:          32768,        0,        0,        0,        0,        0
zio_data_buf_32768:     32768,        0,        0,        0,        0,        0
zio_buf_36864:          36864,        0,        0,        0,        0,        0
zio_data_buf_36864:     36864,        0,        0,        0,        0,        0
zio_buf_40960:          40960,        0,        0,        0,        0,        0
zio_data_buf_40960:     40960,        0,        0,        0,        0,        0
zio_buf_45056:          45056,        0,        0,        0,        0,        0
zio_data_buf_45056:     45056,        0,        0,        0,        0,        0
zio_buf_49152:          49152,        0,        0,        0,        0,        0
zio_data_buf_49152:     49152,        0,        0,        0,        0,        0
zio_buf_53248:          53248,        0,        0,        0,        0,        0
zio_data_buf_53248:     53248,        0,        0,        0,        0,        0
zio_buf_57344:          57344,        0,        0,        0,        0,        0
zio_data_buf_57344:     57344,        0,        0,        0,        0,        0
zio_buf_61440:          61440,        0,        0,        0,        0,        0
zio_data_buf_61440:     61440,        0,        0,        0,        0,        0
zio_buf_65536:          65536,        0,        0,        0,        0,        0
zio_data_buf_65536:     65536,        0,        0,        0,        0,        0
zio_buf_69632:          69632,        0,        0,        0,        0,        0
zio_data_buf_69632:     69632,        0,        0,        0,        0,        0
zio_buf_73728:          73728,        0,        0,        0,        0,        0
zio_data_buf_73728:     73728,        0,        0,        0,        0,        0
zio_buf_77824:          77824,        0,        0,        0,        0,        0
zio_data_buf_77824:     77824,        0,        0,        0,        0,        0
zio_buf_81920:          81920,        0,        0,        0,        0,        0
zio_data_buf_81920:     81920,        0,        0,        0,        0,        0
zio_buf_86016:          86016,        0,        0,        0,        0,        0
zio_data_buf_86016:     86016,        0,        0,        0,        0,        0
zio_buf_90112:          90112,        0,        0,        0,        0,        0
zio_data_buf_90112:     90112,        0,        0,        0,        0,        0
zio_buf_94208:          94208,        0,        0,        0,        0,        0
zio_data_buf_94208:     94208,        0,        0,        0,        0,        0
zio_buf_98304:          98304,        0,        0,        0,        0,        0
zio_data_buf_98304:     98304,        0,        0,        0,        0,        0
zio_buf_102400:        102400,        0,        0,        0,        0,        0
zio_data_buf_102400:   102400,        0,        0,        0,        0,        0
zio_buf_106496:        106496,        0,        0,        0,        0,        0
zio_data_buf_106496:   106496,        0,        0,        0,        0,        0
zio_buf_110592:        110592,        0,        0,        0,        0,        0
zio_data_buf_110592:   110592,        0,        0,        0,        0,        0
zio_buf_114688:        114688,        0,        0,        0,        0,        0
zio_data_buf_114688:   114688,        0,        0,        0,        0,        0
zio_buf_118784:        118784,        0,        0,        0,        0,        0
zio_data_buf_118784:   118784,        0,        0,        0,        0,        0
zio_buf_122880:        122880,        0,        0,        0,        0,        0
zio_data_buf_122880:   122880,        0,        0,        0,        0,        0
zio_buf_126976:        126976,        0,        0,        0,        0,        0
zio_data_buf_126976:   126976,        0,        0,        0,        0,        0
zio_buf_131072:        131072,        0,        0,        0,        0,        0
zio_data_buf_131072:   131072,        0,        0,        0,        0,        0
sa_cache:                  80,        0,     4987,     1988,   176638,        0
dnode_t:                  856,        0,     5815,     2089,    44082,        0
dmu_buf_impl_t:           224,        0,    12748,    36144,  1117594,        0
arc_buf_hdr_t:            216,        0,   229221,      243,  1133714,        0
arc_buf_t:                104,        0,    87053,      787,  1243813,        0
zil_lwb_cache:            192,        0,        2,      778,   236435,        0
zfs_znode_cache:          400,        0,     4987,     2195,   176638,        0
pfsrctrpl:                152,    10000,        0,        0,        0,        0
pfrulepl:                 912,        0,       34,       78,      124,        0
pfstatepl:                392,    10000,       34,     1866,  1123722,        0
pfaltqpl:                 240,        0,        0,        0,        0,        0
pfpooladdrpl:              88,        0,        0,        0,        0,        0
pfrktable:               1296,     1002,        5,       25,       25,        0
pfrkentry:                216,   100008,      658,      890,     2628,        0
pfrkentry2:               216,        0,        0,        0,        0,        0
pffrent:                   32,     5050,        0,        0,        0,        0
pffrag:                    80,        0,        0,        0,        0,        0
pffrcache:                 80,    10035,        0,        0,        0,        0
pffrcent:                  24,    50022,        0,        0,        0,        0
pfstatescrub:              40,        0,        0,        0,        0,        0
pfiaddrpl:                120,        0,        0,        0,        0,        0
pfospfen:                 112,        0,      700,       92,     2800,        0
pfosfp:                    40,        0,      410,      178,     1640,        0

-- 
| Jeremy Chadwick                                jdc at parodius.com |
| Parodius Networking                       http://www.parodius.com/ |
| UNIX Systems Administrator                   Mountain View, CA, US |
| Making life hard for others since 1977.               PGP 4BD6C0CB |




Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20120102125929.GA89966>