From owner-freebsd-stable@freebsd.org Thu Nov 10 08:03:20 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CC3CDC399F7 for ; Thu, 10 Nov 2016 08:03:20 +0000 (UTC) (envelope-from hlh@restart.be) Received: from tignes.restart.be (tignes.restart.be [IPv6:2001:41d0:8:bdbe:0:1::]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "tignes.restart.be", Issuer "CA master" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 66A288C9 for ; Thu, 10 Nov 2016 08:03:20 +0000 (UTC) (envelope-from hlh@restart.be) X-Comment: SPF check N/A for local connections - client-ip=2001:41d0:8:bdbe:1:1::; helo=restart.be; envelope-from=hlh@restart.be; receiver=freebsd-stable@freebsd.org DKIM-Filter: OpenDKIM Filter v2.10.3 tignes.restart.be 3tDwXP5PQSzrGR DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=restart.be; s=tignes; t=1478764997; bh=tbclbwTmJn3nFexFeC/s5KFnpEEjKzhcVsKQynHwBk4=; h=Subject:To:References:From:Date:In-Reply-To; z=Subject:=20Re:=20Freebsd=2011.0=20RELEASE=20-=20ZFS=20deadlock|To :=20freebsd-stable@freebsd.org|References:=20<0c223160-b76f-c635-b b15-4a068ba7efe7@restart.be>=0D=0A=20=0D=0A=20=0D=0A=20<20161109182315.GK794@graf.pompo.net>|From:= 20Henri=20Hennebert=20|Date:=20Thu,=2010=20Nov=202 016=2009:03:16=20+0100|In-Reply-To:=20<20161109182315.GK794@graf.p ompo.net>; b=PB6Q7N4madoZp8Bn1zyysRRQRrcigsqUqitgEG6sy288FCwYMO+DgfvzlQPAD3IGt kTo+fs6rxMoxl3CZNXp8iG1e6oK+9Ii8awjwvDV/HP3IF0wDd6Y6DeUmDBfK8oobAA ggQGcEdRbZiIZslc2u4e/UoiaTxh9+1Nd/JZhehMXKKRxickW4ht7qYxxDXn1p0O04 QU57m0XQwdVjKJwk46weIiZGxXzmhTSn7p6vd1nqDzfTy+Gw7zeSV8VJJf2RpxIU45 Yq0wVPePSpUy7GNxeva/T7VunxHJGyR7/MZTi7Z7SDut1FXhYRTNyTGGnxkgrxR1ph B/xKbU2r48slA== Received: from restart.be (avoriaz.restart.be [IPv6:2001:41d0:8:bdbe:1:1::]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "smtp.restart.be", Issuer "CA master" (verified OK)) by tignes.restart.be (Postfix) with ESMTPS id 3tDwXP5PQSzrGR for ; Thu, 10 Nov 2016 09:03:17 +0100 (CET) Received: from chamonix.restart.bel (chamonix.restart.bel [IPv6:2001:41d0:8:bdbe:1:9:0:0]) (authenticated bits=0) by restart.be (8.15.2/8.15.2) with ESMTPSA id uAA83Ggk026636 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Thu, 10 Nov 2016 09:03:16 +0100 (CET) (envelope-from hlh@restart.be) Subject: Re: Freebsd 11.0 RELEASE - ZFS deadlock To: freebsd-stable@freebsd.org References: <0c223160-b76f-c635-bb15-4a068ba7efe7@restart.be> <20161109182315.GK794@graf.pompo.net> From: Henri Hennebert Message-ID: <840f9bcb-4731-a5a5-a560-2701a7fcf092@restart.be> Date: Thu, 10 Nov 2016 09:03:16 +0100 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:45.0) Gecko/20100101 Thunderbird/45.4.0 MIME-Version: 1.0 In-Reply-To: <20161109182315.GK794@graf.pompo.net> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Nov 2016 08:03:20 -0000 On 11/09/2016 19:23, Thierry Thomas wrote: > Le mer. 9 nov. 16 à 15:03:49 +0100, Henri Hennebert > écrivait : > >> [root@avoriaz ~]# procstat -kk 85656 >> PID TID COMM TDNAME KSTACK >> 85656 101112 find - mi_switch+0xd2 >> sleepq_wait+0x3a sleeplk+0x1b4 __lockmgr_args+0x356 vop_stdlock+0x3c >> VOP_LOCK1_APV+0x8d _vn_lock+0x43 vget+0x47 cache_lookup+0x679 >> vfs_cache_lookup+0xac VOP_LOOKUP_APV+0x87 lookup+0x591 namei+0x572 >> kern_statat+0xa8 sys_fstatat+0x2c amd64_syscall+0x4ce Xfast_syscall+0xfb > > It looks similar to the problem reportes in PR 205163 > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=205163 > > May be causes by too small values for some vfs.zfs.arc*. > Could you please list sysctl for vfs.zfs.arc_max and others? > > Regards, > [root@avoriaz ~]# sysctl vfs.zfs vfs.zfs.trim.max_interval: 1 vfs.zfs.trim.timeout: 30 vfs.zfs.trim.txg_delay: 32 vfs.zfs.trim.enabled: 1 vfs.zfs.vol.unmap_enabled: 1 vfs.zfs.vol.recursive: 0 vfs.zfs.vol.mode: 1 vfs.zfs.version.zpl: 5 vfs.zfs.version.spa: 5000 vfs.zfs.version.acl: 1 vfs.zfs.version.ioctl: 6 vfs.zfs.debug: 0 vfs.zfs.super_owner: 0 vfs.zfs.sync_pass_rewrite: 2 vfs.zfs.sync_pass_dont_compress: 5 vfs.zfs.sync_pass_deferred_free: 2 vfs.zfs.zio.exclude_metadata: 0 vfs.zfs.zio.use_uma: 1 vfs.zfs.cache_flush_disable: 0 vfs.zfs.zil_replay_disable: 0 vfs.zfs.min_auto_ashift: 9 vfs.zfs.max_auto_ashift: 13 vfs.zfs.vdev.trim_max_pending: 10000 vfs.zfs.vdev.bio_delete_disable: 0 vfs.zfs.vdev.bio_flush_disable: 0 vfs.zfs.vdev.write_gap_limit: 4096 vfs.zfs.vdev.read_gap_limit: 32768 vfs.zfs.vdev.aggregation_limit: 131072 vfs.zfs.vdev.trim_max_active: 64 vfs.zfs.vdev.trim_min_active: 1 vfs.zfs.vdev.scrub_max_active: 2 vfs.zfs.vdev.scrub_min_active: 1 vfs.zfs.vdev.async_write_max_active: 10 vfs.zfs.vdev.async_write_min_active: 1 vfs.zfs.vdev.async_read_max_active: 3 vfs.zfs.vdev.async_read_min_active: 1 vfs.zfs.vdev.sync_write_max_active: 10 vfs.zfs.vdev.sync_write_min_active: 10 vfs.zfs.vdev.sync_read_max_active: 10 vfs.zfs.vdev.sync_read_min_active: 10 vfs.zfs.vdev.max_active: 1000 vfs.zfs.vdev.async_write_active_max_dirty_percent: 60 vfs.zfs.vdev.async_write_active_min_dirty_percent: 30 vfs.zfs.vdev.mirror.non_rotating_seek_inc: 1 vfs.zfs.vdev.mirror.non_rotating_inc: 0 vfs.zfs.vdev.mirror.rotating_seek_offset: 1048576 vfs.zfs.vdev.mirror.rotating_seek_inc: 5 vfs.zfs.vdev.mirror.rotating_inc: 0 vfs.zfs.vdev.trim_on_init: 1 vfs.zfs.vdev.cache.bshift: 16 vfs.zfs.vdev.cache.size: 0 vfs.zfs.vdev.cache.max: 16384 vfs.zfs.vdev.metaslabs_per_vdev: 200 vfs.zfs.txg.timeout: 5 vfs.zfs.space_map_blksz: 4096 vfs.zfs.spa_slop_shift: 5 vfs.zfs.spa_asize_inflation: 24 vfs.zfs.deadman_enabled: 1 vfs.zfs.deadman_checktime_ms: 5000 vfs.zfs.deadman_synctime_ms: 1000000 vfs.zfs.debug_flags: 0 vfs.zfs.recover: 0 vfs.zfs.spa_load_verify_data: 1 vfs.zfs.spa_load_verify_metadata: 1 vfs.zfs.spa_load_verify_maxinflight: 10000 vfs.zfs.ccw_retry_interval: 300 vfs.zfs.check_hostid: 1 vfs.zfs.mg_fragmentation_threshold: 85 vfs.zfs.mg_noalloc_threshold: 0 vfs.zfs.condense_pct: 200 vfs.zfs.metaslab.bias_enabled: 1 vfs.zfs.metaslab.lba_weighting_enabled: 1 vfs.zfs.metaslab.fragmentation_factor_enabled: 1 vfs.zfs.metaslab.preload_enabled: 1 vfs.zfs.metaslab.preload_limit: 3 vfs.zfs.metaslab.unload_delay: 8 vfs.zfs.metaslab.load_pct: 50 vfs.zfs.metaslab.min_alloc_size: 33554432 vfs.zfs.metaslab.df_free_pct: 4 vfs.zfs.metaslab.df_alloc_threshold: 131072 vfs.zfs.metaslab.debug_unload: 0 vfs.zfs.metaslab.debug_load: 0 vfs.zfs.metaslab.fragmentation_threshold: 70 vfs.zfs.metaslab.gang_bang: 16777217 vfs.zfs.free_bpobj_enabled: 1 vfs.zfs.free_max_blocks: 18446744073709551615 vfs.zfs.no_scrub_prefetch: 0 vfs.zfs.no_scrub_io: 0 vfs.zfs.resilver_min_time_ms: 3000 vfs.zfs.free_min_time_ms: 1000 vfs.zfs.scan_min_time_ms: 1000 vfs.zfs.scan_idle: 50 vfs.zfs.scrub_delay: 4 vfs.zfs.resilver_delay: 2 vfs.zfs.top_maxinflight: 32 vfs.zfs.zfetch.array_rd_sz: 1048576 vfs.zfs.zfetch.max_distance: 8388608 vfs.zfs.zfetch.min_sec_reap: 2 vfs.zfs.zfetch.max_streams: 8 vfs.zfs.prefetch_disable: 1 vfs.zfs.delay_scale: 500000 vfs.zfs.delay_min_dirty_percent: 60 vfs.zfs.dirty_data_sync: 67108864 vfs.zfs.dirty_data_max_percent: 10 vfs.zfs.dirty_data_max_max: 4294967296 vfs.zfs.dirty_data_max: 373664153 vfs.zfs.max_recordsize: 1048576 vfs.zfs.mdcomp_disable: 0 vfs.zfs.nopwrite_enabled: 1 vfs.zfs.dedup.prefetch: 1 vfs.zfs.l2c_only_size: 0 vfs.zfs.mfu_ghost_data_lsize: 24202240 vfs.zfs.mfu_ghost_metadata_lsize: 136404992 vfs.zfs.mfu_ghost_size: 160607232 vfs.zfs.mfu_data_lsize: 449569280 vfs.zfs.mfu_metadata_lsize: 102724608 vfs.zfs.mfu_size: 714202624 vfs.zfs.mru_ghost_data_lsize: 874834432 vfs.zfs.mru_ghost_metadata_lsize: 387692032 vfs.zfs.mru_ghost_size: 1262526464 vfs.zfs.mru_data_lsize: 151275008 vfs.zfs.mru_metadata_lsize: 13547008 vfs.zfs.mru_size: 322614272 vfs.zfs.anon_data_lsize: 0 vfs.zfs.anon_metadata_lsize: 0 vfs.zfs.anon_size: 2916352 vfs.zfs.l2arc_norw: 1 vfs.zfs.l2arc_feed_again: 1 vfs.zfs.l2arc_noprefetch: 1 vfs.zfs.l2arc_feed_min_ms: 200 vfs.zfs.l2arc_feed_secs: 1 vfs.zfs.l2arc_headroom: 2 vfs.zfs.l2arc_write_boost: 8388608 vfs.zfs.l2arc_write_max: 8388608 vfs.zfs.arc_meta_limit: 402653184 vfs.zfs.arc_free_target: 6204 vfs.zfs.arc_shrink_shift: 7 vfs.zfs.arc_average_blocksize: 8192 vfs.zfs.arc_min: 201326592 vfs.zfs.arc_max: 1610612736 [root@avoriaz ~]# Henri