Date: Tue, 21 Jul 2020 15:37:33 -0400 From: mike tancsa <mike@sentex.net> To: Ronald Klop <ronald-lists@klop.ws>, FreeBSD-STABLE Mailing List <freebsd-stable@freebsd.org> Subject: Re: zfs meta data slowness Message-ID: <975657af-ccac-bbd1-e22b-86270c624226@sentex.net> In-Reply-To: <1949194763.1.1595250243575@localhost> References: <b6d670e6-6e35-a1f7-7daf-76b458f26e30@sentex.net> <1949194763.1.1595250243575@localhost>
next in thread | previous in thread | raw e-mail | index | archive | help
Hi, Thanks for the response. Reply in line On 7/20/2020 9:04 AM, Ronald Klop wrote: > Hi, > > My first suggestion would be to remove a lot of snapshots. But that my > not match your business case. As its a backup server, its sort of the point to have all those snapshots. > Maybe you can provide more information about your setup: > Amount of RAM, CPU? 64G, Xeon(R) CPU E3-1240 v6 @ 3.70GHz > output of "zpool status" # zpool status -x all pools are healthy > output of "zfs list" if possible to share its a big list # zfs list | wc 824 4120 107511 > Type of disks/ssds? old school Device Model: WDC WD80EFAX-68KNBN0 > What is the load of the system? I/O per second, etc. its not cpu bound, disks are sometimes running at 100% based on gstat, but not always > Do you use dedup, GELI? no and no > Something else special about the setup. > output of "top -b" > ports are right now being built in a VM, but the problem (zrepl hanging) and zfs list -t snapshots taking forever happens regardless PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 4439 root 12 40 20 6167M 5762M kqread 3 535:13 200.00% bhyve 98783 root 2 21 0 16M 5136K hdr->b 4 0:01 1.95% zfs 76489 root 21 23 0 738M 54M uwait 1 2:18 0.88% zrepl 98784 root 1 21 0 13M 3832K piperd 3 0:01 0.59% zfs 99563 root 1 20 0 13M 4136K zio->i 4 0:00 0.39% zfs 16136 root 18 25 0 705M 56M uwait 3 29:58 0.00% zrepl-freebsd-amd64 1845 root 1 20 0 12M 3772K nanslp 7 5:54 0.00% ossec-syscheckd 1567 root 1 20 0 11M 2744K select 0 2:22 0.00% syslogd 1737 root 32 20 0 11M 2844K rpcsvc 6 1:40 0.00% nfsd 1660 root 1 -52 r0 11M 11M nanslp 5 1:18 0.00% watchdogd 1434 root 1 20 0 9988K 988K select 3 0:27 0.00% devd 2435 mdtancsa 1 20 0 20M 8008K select 0 0:21 0.00% sshd 1754 root 3 20 0 18M 3556K select 1 0:11 0.00% apcupsd 5917 root 1 20 0 11M 2672K select 2 0:06 0.00% script 1449 _pflogd 1 20 0 12M 3572K bpf 3 0:05 0.00% pflogd ---Mike > That kind of information. > > Regards, > Ronald. > > > Van: mike tancsa <mike@sentex.net> > Datum: zondag, 19 juli 2020 16:17 > Aan: FreeBSD-STABLE Mailing List <freebsd-stable@freebsd.org> > Onderwerp: zfs meta data slowness >> >> Are there any tweaks that can be done to speed up or improve zfs >> metadata performance ? I have a backup server with a lot of snapshots >> (40,000) and just doing a listing can take a great deal of time. Best >> case scenario is about 24 seconds, worst case, I have seen it up to 15 >> minutes. (FreeBSD 12.1-STABLE r363078) >> >> >> ARC Efficiency: 79.33b >> Cache Hit Ratio: 92.81% 73.62b >> Cache Miss Ratio: 7.19% 5.71b >> Actual Hit Ratio: 92.78% 73.60b >> >> Data Demand Efficiency: 96.47% 461.91m >> Data Prefetch Efficiency: 1.00% 262.73m >> >> CACHE HITS BY CACHE LIST: >> Anonymously Used: 0.01% 3.86m >> Most Recently Used: 3.91% 2.88b >> Most Frequently Used: 96.06% 70.72b >> Most Recently Used Ghost: 0.01% 5.31m >> Most Frequently Used Ghost: 0.01% 10.47m >> >> CACHE HITS BY DATA TYPE: >> Demand Data: 0.61% 445.60m >> Prefetch Data: 0.00% 2.63m >> Demand Metadata: 99.36% 73.15b >> Prefetch Metadata: 0.03% 21.00m >> >> CACHE MISSES BY DATA TYPE: >> Demand Data: 0.29% 16.31m >> Prefetch Data: 4.56% 260.10m >> Demand Metadata: 95.02% 5.42b >> Prefetch Metadata: 0.14% 7.75m >> >> >> Other than increase the metadata max, I havent really changed any >> tuneables >> >> >> ZFS Tunables (sysctl): >> kern.maxusers 4416 >> vm.kmem_size 66691842048 >> vm.kmem_size_scale 1 >> vm.kmem_size_min 0 >> vm.kmem_size_max 1319413950874 >> vfs.zfs.trim.max_interval 1 >> vfs.zfs.trim.timeout 30 >> vfs.zfs.trim.txg_delay 32 >> vfs.zfs.trim.enabled 1 >> vfs.zfs.vol.immediate_write_sz 32768 >> vfs.zfs.vol.unmap_sync_enabled 0 >> vfs.zfs.vol.unmap_enabled 1 >> vfs.zfs.vol.recursive 0 >> vfs.zfs.vol.mode 1 >> vfs.zfs.version.zpl 5 >> vfs.zfs.version.spa 5000 >> vfs.zfs.version.acl 1 >> vfs.zfs.version.ioctl 7 >> vfs.zfs.debug 0 >> vfs.zfs.super_owner 0 >> vfs.zfs.immediate_write_sz 32768 >> vfs.zfs.sync_pass_rewrite 2 >> vfs.zfs.sync_pass_dont_compress 5 >> vfs.zfs.sync_pass_deferred_free 2 >> vfs.zfs.zio.dva_throttle_enabled 1 >> vfs.zfs.zio.exclude_metadata 0 >> vfs.zfs.zio.use_uma 1 >> vfs.zfs.zio.taskq_batch_pct 75 >> vfs.zfs.zil_maxblocksize 131072 >> vfs.zfs.zil_slog_bulk 786432 >> vfs.zfs.zil_nocacheflush 0 >> vfs.zfs.zil_replay_disable 0 >> vfs.zfs.cache_flush_disable 0 >> vfs.zfs.standard_sm_blksz 131072 >> vfs.zfs.dtl_sm_blksz 4096 >> vfs.zfs.min_auto_ashift 9 >> vfs.zfs.max_auto_ashift 13 >> vfs.zfs.vdev.trim_max_pending 10000 >> vfs.zfs.vdev.bio_delete_disable 0 >> vfs.zfs.vdev.bio_flush_disable 0 >> vfs.zfs.vdev.def_queue_depth 32 >> vfs.zfs.vdev.queue_depth_pct 1000 >> vfs.zfs.vdev.write_gap_limit 4096 >> vfs.zfs.vdev.read_gap_limit 32768 >> vfs.zfs.vdev.aggregation_limit_non_rotating131072 >> vfs.zfs.vdev.aggregation_limit 1048576 >> vfs.zfs.vdev.initializing_max_active 1 >> vfs.zfs.vdev.initializing_min_active 1 >> vfs.zfs.vdev.removal_max_active 2 >> vfs.zfs.vdev.removal_min_active 1 >> vfs.zfs.vdev.trim_max_active 64 >> vfs.zfs.vdev.trim_min_active 1 >> vfs.zfs.vdev.scrub_max_active 2 >> vfs.zfs.vdev.scrub_min_active 1 >> vfs.zfs.vdev.async_write_max_active 10 >> vfs.zfs.vdev.async_write_min_active 1 >> vfs.zfs.vdev.async_read_max_active 3 >> vfs.zfs.vdev.async_read_min_active 1 >> vfs.zfs.vdev.sync_write_max_active 10 >> vfs.zfs.vdev.sync_write_min_active 10 >> vfs.zfs.vdev.sync_read_max_active 10 >> vfs.zfs.vdev.sync_read_min_active 10 >> vfs.zfs.vdev.max_active 1000 >> vfs.zfs.vdev.async_write_active_max_dirty_percent60 >> vfs.zfs.vdev.async_write_active_min_dirty_percent30 >> vfs.zfs.vdev.mirror.non_rotating_seek_inc1 >> vfs.zfs.vdev.mirror.non_rotating_inc 0 >> vfs.zfs.vdev.mirror.rotating_seek_offset1048576 >> vfs.zfs.vdev.mirror.rotating_seek_inc 5 >> vfs.zfs.vdev.mirror.rotating_inc 0 >> vfs.zfs.vdev.trim_on_init 1 >> vfs.zfs.vdev.cache.bshift 16 >> vfs.zfs.vdev.cache.size 0 >> vfs.zfs.vdev.cache.max 16384 >> vfs.zfs.vdev.validate_skip 0 >> vfs.zfs.vdev.max_ms_shift 34 >> vfs.zfs.vdev.default_ms_shift 29 >> vfs.zfs.vdev.max_ms_count_limit 131072 >> vfs.zfs.vdev.min_ms_count 16 >> vfs.zfs.vdev.default_ms_count 200 >> vfs.zfs.txg.timeout 5 >> vfs.zfs.space_map_ibs 14 >> vfs.zfs.special_class_metadata_reserve_pct25 >> vfs.zfs.user_indirect_is_special 1 >> vfs.zfs.ddt_data_is_special 1 >> vfs.zfs.spa_allocators 4 >> vfs.zfs.spa_min_slop 134217728 >> vfs.zfs.spa_slop_shift 5 >> vfs.zfs.spa_asize_inflation 24 >> vfs.zfs.deadman_enabled 1 >> vfs.zfs.deadman_checktime_ms 5000 >> vfs.zfs.deadman_synctime_ms 1000000 >> vfs.zfs.debugflags 0 >> vfs.zfs.recover 0 >> vfs.zfs.spa_load_verify_data 1 >> vfs.zfs.spa_load_verify_metadata 1 >> vfs.zfs.spa_load_verify_maxinflight 10000 >> vfs.zfs.max_missing_tvds_scan 0 >> vfs.zfs.max_missing_tvds_cachefile 2 >> vfs.zfs.max_missing_tvds 0 >> vfs.zfs.spa_load_print_vdev_tree 0 >> vfs.zfs.ccw_retry_interval 300 >> vfs.zfs.check_hostid 1 >> vfs.zfs.multihost_fail_intervals 10 >> vfs.zfs.multihost_import_intervals 20 >> vfs.zfs.multihost_interval 1000 >> vfs.zfs.mg_fragmentation_threshold 85 >> vfs.zfs.mg_noalloc_threshold 0 >> vfs.zfs.condense_pct 200 >> vfs.zfs.metaslab_sm_blksz 4096 >> vfs.zfs.metaslab.bias_enabled 1 >> vfs.zfs.metaslab.lba_weighting_enabled 1 >> vfs.zfs.metaslab.fragmentation_factor_enabled1 >> vfs.zfs.metaslab.preload_enabled 1 >> vfs.zfs.metaslab.preload_limit 3 >> vfs.zfs.metaslab.unload_delay 8 >> vfs.zfs.metaslab.load_pct 50 >> vfs.zfs.metaslab.min_alloc_size 33554432 >> vfs.zfs.metaslab.df_free_pct 4 >> vfs.zfs.metaslab.df_alloc_threshold 131072 >> vfs.zfs.metaslab.debug_unload 0 >> vfs.zfs.metaslab.debug_load 0 >> vfs.zfs.metaslab.fragmentation_threshold70 >> vfs.zfs.metaslab.force_ganging 16777217 >> vfs.zfs.free_bpobj_enabled 1 >> vfs.zfs.free_max_blocks -1 >> vfs.zfs.zfs_scan_checkpoint_interval 7200 >> vfs.zfs.zfs_scan_legacy 0 >> vfs.zfs.no_scrub_prefetch 0 >> vfs.zfs.no_scrub_io 0 >> vfs.zfs.resilver_min_time_ms 3000 >> vfs.zfs.free_min_time_ms 1000 >> vfs.zfs.scan_min_time_ms 1000 >> vfs.zfs.scan_idle 50 >> vfs.zfs.scrub_delay 4 >> vfs.zfs.resilver_delay 2 >> vfs.zfs.zfetch.array_rd_sz 1048576 >> vfs.zfs.zfetch.max_idistance 67108864 >> vfs.zfs.zfetch.max_distance 8388608 >> vfs.zfs.zfetch.min_sec_reap 2 >> vfs.zfs.zfetch.max_streams 8 >> vfs.zfs.prefetch_disable 0 >> vfs.zfs.delay_scale 500000 >> vfs.zfs.delay_min_dirty_percent 60 >> vfs.zfs.dirty_data_sync_pct 20 >> vfs.zfs.dirty_data_max_percent 10 >> vfs.zfs.dirty_data_max_max 4294967296 >> vfs.zfs.dirty_data_max 4294967296 >> vfs.zfs.max_recordsize 1048576 >> vfs.zfs.default_ibs 17 >> vfs.zfs.default_bs 9 >> vfs.zfs.send_holes_without_birth_time 1 >> vfs.zfs.mdcomp_disable 0 >> vfs.zfs.per_txg_dirty_frees_percent 5 >> vfs.zfs.nopwrite_enabled 1 >> vfs.zfs.dedup.prefetch 1 >> vfs.zfs.dbuf_cache_lowater_pct 10 >> vfs.zfs.dbuf_cache_hiwater_pct 10 >> vfs.zfs.dbuf_metadata_cache_overflow 0 >> vfs.zfs.dbuf_metadata_cache_shift 6 >> vfs.zfs.dbuf_cache_shift 5 >> vfs.zfs.dbuf_metadata_cache_max_bytes 1025282816 >> vfs.zfs.dbuf_cache_max_bytes 2050565632 >> vfs.zfs.arc_min_prescient_prefetch_ms 6 >> vfs.zfs.arc_min_prefetch_ms 1 >> vfs.zfs.l2c_only_size 0 >> vfs.zfs.mfu_ghost_data_esize 7778263552 >> vfs.zfs.mfu_ghost_metadata_esize 16851792896 >> vfs.zfs.mfu_ghost_size 24630056448 >> vfs.zfs.mfu_data_esize 3059418112 >> vfs.zfs.mfu_metadata_esize 28641792 >> vfs.zfs.mfu_size 6399023104 >> vfs.zfs.mru_ghost_data_esize 2199812096 >> vfs.zfs.mru_ghost_metadata_esize 6289682432 >> vfs.zfs.mru_ghost_size 8489494528 >> vfs.zfs.mru_data_esize 22781456384 >> vfs.zfs.mru_metadata_esize 309155840 >> vfs.zfs.mru_size 23847875584 >> vfs.zfs.anon_data_esize 0 >> vfs.zfs.anon_metadata_esize 0 >> vfs.zfs.anon_size 8556544 >> vfs.zfs.l2arc_norw 1 >> vfs.zfs.l2arc_feed_again 1 >> vfs.zfs.l2arc_noprefetch 1 >> vfs.zfs.l2arc_feed_min_ms 200 >> vfs.zfs.l2arc_feed_secs 1 >> vfs.zfs.l2arc_headroom 2 >> vfs.zfs.l2arc_write_boost 8388608 >> vfs.zfs.l2arc_write_max 8388608 >> vfs.zfs.arc_meta_strategy 1 >> vfs.zfs.arc_meta_limit 15833624576 >> vfs.zfs.arc_free_target 346902 >> vfs.zfs.arc_kmem_cache_reap_retry_ms 1000 >> vfs.zfs.compressed_arc_enabled 1 >> vfs.zfs.arc_grow_retry 60 >> vfs.zfs.arc_shrink_shift 7 >> vfs.zfs.arc_average_blocksize 8192 >> vfs.zfs.arc_no_grow_shift 5 >> vfs.zfs.arc_min 8202262528 >> vfs.zfs.arc_max 39334498304 >> vfs.zfs.abd_chunk_size 4096 >> vfs.zfs.abd_scatter_enabled 1 >> >> _______________________________________________ >> freebsd-stable@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-stable >> To unsubscribe, send any mail to >> "freebsd-stable-unsubscribe@freebsd.org" >> >> >> > _______________________________________________ > freebsd-stable@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?975657af-ccac-bbd1-e22b-86270c624226>