From owner-freebsd-stable@FreeBSD.ORG Mon Jan 2 12:59:32 2012 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 964A7106567E for ; Mon, 2 Jan 2012 12:59:32 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta01.westchester.pa.mail.comcast.net (qmta02.westchester.pa.mail.comcast.net [76.96.62.24]) by mx1.freebsd.org (Postfix) with ESMTP id 2E13F8FC0A for ; Mon, 2 Jan 2012 12:59:31 +0000 (UTC) Received: from omta20.westchester.pa.mail.comcast.net ([76.96.62.71]) by qmta01.westchester.pa.mail.comcast.net with comcast id Gcqk1i0031YDfWL51czYDq; Mon, 02 Jan 2012 12:59:32 +0000 Received: from koitsu.dyndns.org ([67.180.84.87]) by omta20.westchester.pa.mail.comcast.net with comcast id GczW1i0191t3BNj3gczX2A; Mon, 02 Jan 2012 12:59:32 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id 39707102C1E; Mon, 2 Jan 2012 04:59:29 -0800 (PST) Date: Mon, 2 Jan 2012 04:59:29 -0800 From: Jeremy Chadwick To: Victor Balada Diaz Message-ID: <20120102125929.GA89966@icarus.home.lan> References: <20120102122549.GB17398@equilibrium.bsdes.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20120102122549.GB17398@equilibrium.bsdes.net> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-stable@freebsd.org Subject: Re: Performance problems with pagedaemon X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 Jan 2012 12:59:32 -0000 On Mon, Jan 02, 2012 at 01:25:49PM +0100, Victor Balada Diaz wrote: > Hello, > > First: Happy new year to everybody! > > Last year i had a problem with pagedaemon that i reported here[1] but i had > no replies. The problem went away and now it's back to beat me again. > > Seems that mysql creates some bad interactions with pagedaemon and system starts > to get slow for disk access. Eg, accessing a man page takes a few seconds. If > i stop mysql and start it again, it recovers again. As this is a problem that have > appeared just twice (on October and now), it's not easy to reproduce and test. > > Disk I/O as per gstat is not high, not many context switches, etc. Bottleneck > seems to be CPU. I've reported more information on october's mail[1]. > > What can i do to debug it? Seems that maybe the most interesting thing would be > enabling lock profiling and wait for the system to fail again, but is there anything > that i can do/report that would be helpful to track the problem down now that's > failing? > > Thanks a lot. > Regards. > > [1]: http://lists.freebsd.org/pipermail/freebsd-stable/2011-October/064248.html The only things I can think of are output from the following: * vmstat -z * sysctl -a vm * top (mainly for mysqld) * top -H -S Is this machine in question still running 8.1-RELEASE? If so, have you tried upgrading it to 8.2-STABLE? I believe there have been some VM adjustments, but I'd have to go back and examine commits. Taken from our mysql server. Note that the mysql storage filesystem is stored on ZFS as well, which has added benefits/complexities yet we see no issue. FreeBSD XXXXXXXXXXXXXXXXX 8.2-STABLE FreeBSD 8.2-STABLE #0: Tue Dec 13 00:00:16 PST 2011 root@XXXXXXXXXXXXXXXXX:/usr/obj/usr/src/sys/X7SBI_RELENG_8_amd64 amd64 real memory = 8589934592 (8192 MB) avail memory = 8250040320 (7867 MB) Mem: 539M Active, 2559M Inact, 3218M Wired, 4K Cache, 828M Buf, 1603M Free Swap: 16G Total, 16G Free PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 1940 mysql 16 20 0 781M 401M sigwai 1 158:40 0.00% [mysqld] top -H -S shows no indication pagedaemon is "going crazy". loader.conf tunings: # Increase maximum allocatable memory on a process to 2.5GB. # We don't choose 8GB (our amount of RAM) since that would # exhaust all memory, and result in a kernel panic. Maximum # stack size is set to 256MB. One can view these settings # using limits(1). # # dfldsiz = Initial data size limit (bytes) # maxdsiz = Maximum data size limit (bytes) # dflssiz = Initial stack size limit (bytes) # maxssiz = Maximum stack size limit (bytes) # kern.maxdsiz="2560M" kern.dfldsiz="2560M" kern.maxssiz="256M" # Limit ZFS ARC maximum. # NOTE #1: In 8.2-RELEASE and onward, vm.kmem_size_scale defaults to 1, # which means vm.kmem_size should match the amount of RAM installed # in the system. If using an earlier FreeBSD release, be sure to set # vm.kmem_size manually to the amount of RAM you have. # NOTE #2: Do not set vm.kmem_size to 2x that of physical RAM, otherwise # vfs.zfs.arc_max effectively becomes halved. # http://lists.freebsd.org/pipermail/freebsd-fs/2011-March/010875.html vfs.zfs.arc_max="5120M" my.cnf tunings: [mysqld] ; http://dev.mysql.com/doc/refman/5.5/en/table-cache.html table_open_cache = 2048 max_connections = 512 tmp_table_size = 64M max_allowed_packet = 32M key_buffer_size = 64M join_buffer_size = 16M sort_buffer_size = 16M read_buffer_size = 16M query_cache_size = 64M query_cache_limit = 32M innodb_file_per_table innodb_buffer_pool_size = 512M innodb_log_file_size = 128M innodb_log_buffer_size = 8M innodb_additional_mem_pool_size = 20M sysctl -a vm output: vm.vmtotal: System wide totals computed every five seconds: (values in kilobytes) =============================================== Processes: (RUNQ: 1 Disk Wait: 1 Page Wait: 0 Sleep: 59) Virtual Memory: (Total: 1075323828K Active: 1483408K) Real Memory: (Total: 2970948K Active: 585264K) Shared Virtual Memory: (Total: 17884K Active: 6756K) Shared Real Memory: (Total: 7500K Active: 6564K) Free Memory Pages: 1641708K vm.loadavg: { 0.00 0.00 0.00 } vm.v_free_min: 12824 vm.v_free_target: 53985 vm.v_free_reserved: 2689 vm.v_inactive_target: 80977 vm.v_cache_min: 53985 vm.v_cache_max: 107970 vm.v_pageout_free_min: 34 vm.pageout_algorithm: 0 vm.swap_enabled: 1 vm.md_malloc_wait: 0 vm.kmem_map_free: 2897080320 vm.kmem_map_size: 2349465600 vm.kmem_size_scale: 1 vm.kmem_size_max: 329853485875 vm.kmem_size_min: 0 vm.kmem_size: 8303726592 vm.nswapdev: 1 vm.dmmax: 32 vm.swap_async_max: 4 vm.overcommit: 0 vm.swap_reserved: 11426652160 vm.swap_total: 17179869184 vm.zone_count: 195 vm.swap_idle_threshold2: 10 vm.swap_idle_threshold1: 2 vm.kstacks: 475 vm.kstack_cache_size: 128 vm.exec_map_entries: 16 vm.stats.misc.zero_page_count: 158 vm.stats.misc.cnt_prezero: 0 vm.stats.vm.v_kthreadpages: 0 vm.stats.vm.v_rforkpages: 0 vm.stats.vm.v_vforkpages: 35572524 vm.stats.vm.v_forkpages: 131458827 vm.stats.vm.v_kthreads: 22 vm.stats.vm.v_rforks: 0 vm.stats.vm.v_vforks: 159338 vm.stats.vm.v_forks: 711839 vm.stats.vm.v_interrupt_free_min: 2 vm.stats.vm.v_pageout_free_min: 34 vm.stats.vm.v_cache_max: 107970 vm.stats.vm.v_cache_min: 53985 vm.stats.vm.v_cache_count: 1 vm.stats.vm.v_inactive_count: 655146 vm.stats.vm.v_inactive_target: 80977 vm.stats.vm.v_active_count: 137837 vm.stats.vm.v_wire_count: 823816 vm.stats.vm.v_free_count: 410409 vm.stats.vm.v_free_min: 12824 vm.stats.vm.v_free_target: 53985 vm.stats.vm.v_free_reserved: 2689 vm.stats.vm.v_page_count: 2027277 vm.stats.vm.v_page_size: 4096 vm.stats.vm.v_tfree: 488602276 vm.stats.vm.v_pfree: 167595871 vm.stats.vm.v_dfree: 0 vm.stats.vm.v_tcached: 42510 vm.stats.vm.v_pdpages: 0 vm.stats.vm.v_pdwakeups: 0 vm.stats.vm.v_reactivated: 8878 vm.stats.vm.v_intrans: 19805 vm.stats.vm.v_vnodepgsout: 1 vm.stats.vm.v_vnodepgsin: 58873 vm.stats.vm.v_vnodeout: 1 vm.stats.vm.v_vnodein: 47194 vm.stats.vm.v_swappgsout: 0 vm.stats.vm.v_swappgsin: 0 vm.stats.vm.v_swapout: 0 vm.stats.vm.v_swapin: 0 vm.stats.vm.v_ozfod: 511 vm.stats.vm.v_zfod: 231124379 vm.stats.vm.v_cow_optim: 41799 vm.stats.vm.v_cow_faults: 39666798 vm.stats.vm.v_vm_faults: 299438319 vm.stats.sys.v_soft: 411033343 vm.stats.sys.v_intr: 263204059 vm.stats.sys.v_syscall: 3214862746 vm.stats.sys.v_trap: 645468188 vm.stats.sys.v_swtch: 2209989736 vm.stats.object.bypasses: 524363 vm.stats.object.collapses: 3306408 vm.v_free_severe: 7756 vm.old_msync: 0 vm.msync_flush_flags: 0 vm.boot_pages: 48 vm.max_wired: 666530 vm.pageout_lock_miss: 0 vm.disable_swapspace_pageouts: 0 vm.defer_swapspace_pageouts: 0 vm.swap_idle_enabled: 0 vm.pageout_stats_interval: 5 vm.pageout_full_stats_interval: 20 vm.pageout_stats_max: 53985 vm.max_launder: 32 vm.phys_segs: SEGMENT 0: start: 0x1000 end: 0x97000 free list: 0xffffffff80718da8 SEGMENT 1: start: 0x8ad000 end: 0x1000000 free list: 0xffffffff80718da8 SEGMENT 2: start: 0x1000000 end: 0xcfe70000 free list: 0xffffffff80718a00 SEGMENT 3: start: 0x100000000 end: 0x21f8b4000 free list: 0xffffffff80718a00 vm.phys_free: FREE LIST 0: ORDER (SIZE) | NUMBER | POOL 0 | POOL 1 | POOL 2 -- -- -- -- -- -- -- -- 12 ( 16384K) | 0 | 0 | 0 11 ( 8192K) | 0 | 0 | 0 10 ( 4096K) | 22 | 0 | 0 9 ( 2048K) | 74 | 0 | 0 8 ( 1024K) | 10 | 0 | 0 7 ( 512K) | 15 | 1 | 0 6 ( 256K) | 10 | 1 | 0 5 ( 128K) | 4 | 1 | 0 4 ( 64K) | 1 | 0 | 0 3 ( 32K) | 1 | 0 | 0 2 ( 16K) | 0 | 0 | 0 1 ( 8K) | 0 | 1 | 0 0 ( 4K) | 0 | 1 | 0 FREE LIST 1: ORDER (SIZE) | NUMBER | POOL 0 | POOL 1 | POOL 2 -- -- -- -- -- -- -- -- 12 ( 16384K) | 0 | 0 | 0 11 ( 8192K) | 0 | 0 | 0 10 ( 4096K) | 0 | 0 | 0 9 ( 2048K) | 0 | 0 | 0 8 ( 1024K) | 0 | 0 | 0 7 ( 512K) | 0 | 0 | 0 6 ( 256K) | 0 | 0 | 0 5 ( 128K) | 0 | 0 | 0 4 ( 64K) | 0 | 0 | 0 3 ( 32K) | 0 | 0 | 0 2 ( 16K) | 1 | 0 | 0 1 ( 8K) | 0 | 0 | 0 0 ( 4K) | 0 | 0 | 0 vm.reserv.reclaimed: 246 vm.reserv.partpopq: LEVEL SIZE NUMBER -1: 1377964K, 1226 vm.reserv.freed: 2823528 vm.reserv.broken: 5973 vm.idlezero_enable: 0 vm.kvm_free: 539964207104 vm.kvm_size: 549755809792 vm.pmap.pmap_collect_active: 0 vm.pmap.pmap_collect_inactive: 0 vm.pmap.pv_entry_spare: 5609 vm.pmap.pv_entry_allocs: 932709470 vm.pmap.pv_entry_frees: 932651743 vm.pmap.pc_chunk_tryfail: 0 vm.pmap.pc_chunk_frees: 5562480 vm.pmap.pc_chunk_allocs: 5562857 vm.pmap.pc_chunk_count: 377 vm.pmap.pv_entry_count: 57727 vm.pmap.pdpe.demotions: 0 vm.pmap.pde.promotions: 3234381 vm.pmap.pde.p_failures: 36313949 vm.pmap.pde.mappings: 10185 vm.pmap.pde.demotions: 3202630 vm.pmap.shpgperproc: 200 vm.pmap.pv_entry_max: 3260077 vm.pmap.pg_ps_enabled: 1 vm.pmap.pat_works: 1 vmstat -z output: ITEM SIZE LIMIT USED FREE REQUESTS FAILURES UMA Kegs: 208, 0, 193, 11, 193, 0 UMA Zones: 320, 0, 193, 11, 193, 0 UMA Slabs: 568, 0, 91344, 573, 22063422, 0 UMA RCntSlabs: 568, 0, 1652, 0, 1652, 0 UMA Hash: 256, 0, 79, 11, 81, 0 16 Bucket: 152, 0, 198, 2, 198, 0 32 Bucket: 280, 0, 220, 4, 220, 0 64 Bucket: 536, 0, 237, 1, 237, 324 128 Bucket: 1048, 0, 5090, 1, 5090, 3812 VM OBJECT: 216, 0, 112042, 1682, 19218174, 0 MAP: 232, 0, 7, 25, 7, 0 KMAP ENTRY: 120, 278070, 2358, 8802, 56999587, 0 MAP ENTRY: 120, 0, 1254, 2063, 33749125, 0 DP fakepg: 120, 0, 0, 0, 0, 0 SG fakepg: 120, 0, 0, 0, 0, 0 mt_zone: 2056, 0, 203, 34, 203, 0 16: 16, 0, 5939, 4309, 812209016, 0 32: 32, 0, 2778, 9948, 36156302, 0 64: 64, 0, 15479, 124465, 559617065, 0 128: 128, 0, 5998, 6124, 500883081, 0 256: 256, 0, 1542, 105408, 116395928, 0 512: 512, 0, 5900, 5195, 30945149, 0 1024: 1024, 0, 203, 1789, 10984378, 0 2048: 2048, 0, 449, 4339, 14614660, 0 4096: 4096, 0, 789, 784, 11198444, 0 Files: 80, 0, 780, 795, 19941086, 0 TURNSTILE: 136, 0, 778, 102, 778, 0 umtx pi: 96, 0, 0, 0, 0, 0 MAC labels: 40, 0, 0, 0, 0, 0 PROC: 1136, 0, 62, 181, 871274, 0 THREAD: 1096, 0, 475, 302, 1185892, 0 SLEEPQUEUE: 80, 0, 778, 150, 778, 0 VMSPACE: 392, 0, 41, 189, 871256, 0 cpuset: 72, 0, 2, 98, 2, 0 audit_record: 952, 0, 0, 0, 0, 0 mbuf_packet: 256, 0, 2046, 770, 119097913, 0 mbuf: 256, 0, 2, 902, 596890710, 0 mbuf_cluster: 2048, 25600, 2816, 116, 2834, 0 mbuf_jumbo_page: 4096, 12800, 0, 186, 8204717, 0 mbuf_jumbo_9k: 9216, 6400, 0, 0, 0, 0 mbuf_jumbo_16k: 16384, 3200, 0, 0, 0, 0 mbuf_ext_refcnt: 4, 0, 0, 0, 0, 0 g_bio: 232, 0, 0, 12624, 132751934, 0 ttyinq: 160, 0, 225, 207, 1260, 0 ttyoutq: 256, 0, 117, 153, 663, 0 ata_request: 320, 0, 0, 84, 458, 0 ata_composite: 336, 0, 0, 0, 0, 0 VNODE: 472, 0, 162386, 758, 7441920, 0 VNODEPOLL: 112, 0, 0, 0, 0, 0 NAMEI: 1024, 0, 0, 96, 56473208, 0 S VFS Cache: 108, 0, 178642, 647, 8950044, 0 L VFS Cache: 328, 0, 860, 2428, 128661, 0 DIRHASH: 1024, 0, 930, 134, 1315, 0 NFSMOUNT: 632, 0, 1, 11, 1, 0 NFSNODE: 688, 0, 6, 19, 6, 0 pipe: 728, 0, 40, 115, 292578, 0 ksiginfo: 112, 0, 232, 824, 86359, 0 itimer: 344, 0, 1, 54, 6, 0 KNOTE: 128, 0, 137, 414, 13517474, 0 socket: 680, 25602, 114, 180, 1932153, 0 unpcb: 240, 25600, 49, 207, 121492, 0 ipq: 56, 819, 0, 0, 0, 0 udp_inpcb: 336, 25608, 19, 168, 351143, 0 udpcb: 16, 25704, 19, 821, 351143, 0 tcp_inpcb: 336, 25608, 49, 633, 1226932, 0 tcpcb: 944, 25600, 36, 124, 1226932, 0 tcptw: 72, 5150, 13, 837, 479105, 0 syncache: 144, 15366, 0, 156, 1191373, 0 hostcache: 136, 15372, 2, 166, 267, 0 tcpreass: 40, 1680, 0, 336, 703, 0 sackhole: 32, 0, 0, 505, 725, 0 ripcb: 336, 25608, 8, 146, 232583, 0 rtentry: 200, 0, 12, 64, 12, 0 selfd: 56, 0, 257, 751, 629664344, 0 SWAPMETA: 288, 116519, 0, 0, 0, 0 ip4flow: 56, 99351, 0, 0, 0, 0 ip6flow: 80, 99360, 0, 0, 0, 0 Mountpoints: 752, 0, 9, 16, 9, 0 FFS inode: 168, 0, 157347, 4925, 7264635, 0 FFS1 dinode: 128, 0, 0, 0, 0, 0 FFS2 dinode: 256, 0, 157347, 4848, 7264635, 0 taskq_zone: 48, 0, 0, 1008, 392677, 0 zio_cache: 880, 0, 1, 23083, 693486370, 0 zio_link_cache: 48, 0, 0, 23472, 267139845, 0 zio_buf_512: 512, 0, 0, 0, 0, 0 zio_data_buf_512: 512, 0, 0, 0, 0, 0 zio_buf_1024: 1024, 0, 0, 0, 0, 0 zio_data_buf_1024: 1024, 0, 0, 0, 0, 0 zio_buf_1536: 1536, 0, 0, 0, 0, 0 zio_data_buf_1536: 1536, 0, 0, 0, 0, 0 zio_buf_2048: 2048, 0, 0, 0, 0, 0 zio_data_buf_2048: 2048, 0, 0, 0, 0, 0 zio_buf_2560: 2560, 0, 0, 0, 0, 0 zio_data_buf_2560: 2560, 0, 0, 0, 0, 0 zio_buf_3072: 3072, 0, 0, 0, 0, 0 zio_data_buf_3072: 3072, 0, 0, 0, 0, 0 zio_buf_3584: 3584, 0, 0, 0, 0, 0 zio_data_buf_3584: 3584, 0, 0, 0, 0, 0 zio_buf_4096: 4096, 0, 0, 0, 0, 0 zio_data_buf_4096: 4096, 0, 0, 0, 0, 0 zio_buf_5120: 5120, 0, 0, 0, 0, 0 zio_data_buf_5120: 5120, 0, 0, 0, 0, 0 zio_buf_6144: 6144, 0, 0, 0, 0, 0 zio_data_buf_6144: 6144, 0, 0, 0, 0, 0 zio_buf_7168: 7168, 0, 0, 0, 0, 0 zio_data_buf_7168: 7168, 0, 0, 0, 0, 0 zio_buf_8192: 8192, 0, 0, 0, 0, 0 zio_data_buf_8192: 8192, 0, 0, 0, 0, 0 zio_buf_10240: 10240, 0, 0, 0, 0, 0 zio_data_buf_10240: 10240, 0, 0, 0, 0, 0 zio_buf_12288: 12288, 0, 0, 0, 0, 0 zio_data_buf_12288: 12288, 0, 0, 0, 0, 0 zio_buf_14336: 14336, 0, 0, 0, 0, 0 zio_data_buf_14336: 14336, 0, 0, 0, 0, 0 zio_buf_16384: 16384, 0, 0, 0, 0, 0 zio_data_buf_16384: 16384, 0, 0, 0, 0, 0 zio_buf_20480: 20480, 0, 0, 0, 0, 0 zio_data_buf_20480: 20480, 0, 0, 0, 0, 0 zio_buf_24576: 24576, 0, 0, 0, 0, 0 zio_data_buf_24576: 24576, 0, 0, 0, 0, 0 zio_buf_28672: 28672, 0, 0, 0, 0, 0 zio_data_buf_28672: 28672, 0, 0, 0, 0, 0 zio_buf_32768: 32768, 0, 0, 0, 0, 0 zio_data_buf_32768: 32768, 0, 0, 0, 0, 0 zio_buf_36864: 36864, 0, 0, 0, 0, 0 zio_data_buf_36864: 36864, 0, 0, 0, 0, 0 zio_buf_40960: 40960, 0, 0, 0, 0, 0 zio_data_buf_40960: 40960, 0, 0, 0, 0, 0 zio_buf_45056: 45056, 0, 0, 0, 0, 0 zio_data_buf_45056: 45056, 0, 0, 0, 0, 0 zio_buf_49152: 49152, 0, 0, 0, 0, 0 zio_data_buf_49152: 49152, 0, 0, 0, 0, 0 zio_buf_53248: 53248, 0, 0, 0, 0, 0 zio_data_buf_53248: 53248, 0, 0, 0, 0, 0 zio_buf_57344: 57344, 0, 0, 0, 0, 0 zio_data_buf_57344: 57344, 0, 0, 0, 0, 0 zio_buf_61440: 61440, 0, 0, 0, 0, 0 zio_data_buf_61440: 61440, 0, 0, 0, 0, 0 zio_buf_65536: 65536, 0, 0, 0, 0, 0 zio_data_buf_65536: 65536, 0, 0, 0, 0, 0 zio_buf_69632: 69632, 0, 0, 0, 0, 0 zio_data_buf_69632: 69632, 0, 0, 0, 0, 0 zio_buf_73728: 73728, 0, 0, 0, 0, 0 zio_data_buf_73728: 73728, 0, 0, 0, 0, 0 zio_buf_77824: 77824, 0, 0, 0, 0, 0 zio_data_buf_77824: 77824, 0, 0, 0, 0, 0 zio_buf_81920: 81920, 0, 0, 0, 0, 0 zio_data_buf_81920: 81920, 0, 0, 0, 0, 0 zio_buf_86016: 86016, 0, 0, 0, 0, 0 zio_data_buf_86016: 86016, 0, 0, 0, 0, 0 zio_buf_90112: 90112, 0, 0, 0, 0, 0 zio_data_buf_90112: 90112, 0, 0, 0, 0, 0 zio_buf_94208: 94208, 0, 0, 0, 0, 0 zio_data_buf_94208: 94208, 0, 0, 0, 0, 0 zio_buf_98304: 98304, 0, 0, 0, 0, 0 zio_data_buf_98304: 98304, 0, 0, 0, 0, 0 zio_buf_102400: 102400, 0, 0, 0, 0, 0 zio_data_buf_102400: 102400, 0, 0, 0, 0, 0 zio_buf_106496: 106496, 0, 0, 0, 0, 0 zio_data_buf_106496: 106496, 0, 0, 0, 0, 0 zio_buf_110592: 110592, 0, 0, 0, 0, 0 zio_data_buf_110592: 110592, 0, 0, 0, 0, 0 zio_buf_114688: 114688, 0, 0, 0, 0, 0 zio_data_buf_114688: 114688, 0, 0, 0, 0, 0 zio_buf_118784: 118784, 0, 0, 0, 0, 0 zio_data_buf_118784: 118784, 0, 0, 0, 0, 0 zio_buf_122880: 122880, 0, 0, 0, 0, 0 zio_data_buf_122880: 122880, 0, 0, 0, 0, 0 zio_buf_126976: 126976, 0, 0, 0, 0, 0 zio_data_buf_126976: 126976, 0, 0, 0, 0, 0 zio_buf_131072: 131072, 0, 0, 0, 0, 0 zio_data_buf_131072: 131072, 0, 0, 0, 0, 0 sa_cache: 80, 0, 4987, 1988, 176638, 0 dnode_t: 856, 0, 5815, 2089, 44082, 0 dmu_buf_impl_t: 224, 0, 12748, 36144, 1117594, 0 arc_buf_hdr_t: 216, 0, 229221, 243, 1133714, 0 arc_buf_t: 104, 0, 87053, 787, 1243813, 0 zil_lwb_cache: 192, 0, 2, 778, 236435, 0 zfs_znode_cache: 400, 0, 4987, 2195, 176638, 0 pfsrctrpl: 152, 10000, 0, 0, 0, 0 pfrulepl: 912, 0, 34, 78, 124, 0 pfstatepl: 392, 10000, 34, 1866, 1123722, 0 pfaltqpl: 240, 0, 0, 0, 0, 0 pfpooladdrpl: 88, 0, 0, 0, 0, 0 pfrktable: 1296, 1002, 5, 25, 25, 0 pfrkentry: 216, 100008, 658, 890, 2628, 0 pfrkentry2: 216, 0, 0, 0, 0, 0 pffrent: 32, 5050, 0, 0, 0, 0 pffrag: 80, 0, 0, 0, 0, 0 pffrcache: 80, 10035, 0, 0, 0, 0 pffrcent: 24, 50022, 0, 0, 0, 0 pfstatescrub: 40, 0, 0, 0, 0, 0 pfiaddrpl: 120, 0, 0, 0, 0, 0 pfospfen: 112, 0, 700, 92, 2800, 0 pfosfp: 40, 0, 410, 178, 1640, 0 -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB |