From owner-freebsd-fs@freebsd.org Tue Dec 22 15:24:46 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0FDAAA4F52F for ; Tue, 22 Dec 2015 15:24:46 +0000 (UTC) (envelope-from freebsd-listen@fabiankeil.de) Received: from smtprelay05.ispgateway.de (smtprelay05.ispgateway.de [80.67.31.99]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 63E8B11D3 for ; Tue, 22 Dec 2015 15:24:45 +0000 (UTC) (envelope-from freebsd-listen@fabiankeil.de) Received: from [78.35.157.50] (helo=fabiankeil.de) by smtprelay05.ispgateway.de with esmtpsa (TLSv1.2:AES128-GCM-SHA256:128) (Exim 4.84) (envelope-from ) id 1aBOfD-0000ZH-LF for freebsd-fs@freebsd.org; Tue, 22 Dec 2015 16:15:47 +0100 Date: Tue, 22 Dec 2015 16:12:00 +0100 From: Fabian Keil To: FreeBSD FS Subject: ZFS:dmu_objset_find_dp_impl() - panic: vm_fault: fault on nofault entry, addr: fffffe0094653000 Message-ID: <20151222161200.19ab1832@fabiankeil.de> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; boundary="Sig_/I1kt9lqEZfT7evbjkYjI0DA"; protocol="application/pgp-signature" X-Df-Sender: Nzc1MDY3 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 22 Dec 2015 15:24:46 -0000 --Sig_/I1kt9lqEZfT7evbjkYjI0DA Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Using a kernel based on r292334, I got this panic while importing a ZFS pool with vfs.zfs.spa_load_verify_data and vfs.zfs.spa_load_verify_metadata set to 0. I've not been able to reproduce it yet and the changed sysctl's above may not actually matter (but I usually use the defaults). The pool has a single leaf vdev that is backed by ggatec which transfers the data over a slow and easily saturated connection (< ~120 kB/s up). Graph: https://www.fabiankeil.de/talks/versteckter-block-speicher/mgp00030.html fk@r500 /usr/crash $kgdb /usr/lib/debug/boot/kernel/kernel.debug vmcore.2=20 [...] Unread portion of the kernel message buffer: [11912] panic: vm_fault: fault on nofault entry, addr: fffffe0094653000 [11912] cpuid =3D 0 [11912] KDB: stack backtrace: [...] #0 doadump (textdump=3D0) at pcpu.h:221 221 pcpu.h: No such file or directory. in pcpu.h (kgdb) where #0 doadump (textdump=3D0) at pcpu.h:221 #1 0xffffffff8031752b in db_dump (dummy=3D, dummy2=3D= false, dummy3=3D0, dummy4=3D0x0) at /usr/src/sys/ddb/db_command.c:533 #2 0xffffffff8031731e in db_command (cmd_table=3D0x0) at /usr/src/sys/ddb/= db_command.c:440 #3 0xffffffff803170b4 in db_command_loop () at /usr/src/sys/ddb/db_command= .c:493 #4 0xffffffff80319bbb in db_trap (type=3D, code=3D0) = at /usr/src/sys/ddb/db_main.c:251 #5 0xffffffff805e2dc3 in kdb_trap (type=3D3, code=3D0, tf=3D) at /usr/src/sys/kern/subr_kdb.c:654 #6 0xffffffff8087f207 in trap (frame=3D0xfffffe0094f8f220) at /usr/src/sys= /amd64/amd64/trap.c:549 #7 0xffffffff808641b7 in calltrap () at /usr/src/sys/amd64/amd64/exception= .S:234 #8 0xffffffff805e24ab in kdb_enter (why=3D0xffffffff8097216b "panic", msg= =3D0x32
) at cpufunc.h:63 #9 0xffffffff8059ea4f in vpanic (fmt=3D, ap=3D) at /usr/src/sys/kern/kern_shutdown.c:750 #10 0xffffffff8059e8a3 in panic (fmt=3D0x0) at /usr/src/sys/kern/kern_shutd= own.c:688 #11 0xffffffff80835650 in vm_fault_hold (map=3D, vaddr= =3D, fault_type=3D, fault_flags= =3D, m_hold=3D) at /usr/src/sys/vm/vm_fault.c:332 #12 0xffffffff808332f8 in vm_fault (map=3D0xfffff80002000000, vaddr=3D, fault_type=3D1 '\001', fault_flags=3D0) at /usr/src/sys/v= m/vm_fault.c:277 #13 0xffffffff8087f97a in trap_pfault (frame=3D0xfffffe0094f8f8d0, usermode= =3D0) at /usr/src/sys/amd64/amd64/trap.c:734 #14 0xffffffff8087f21e in trap (frame=3D0xfffffe0094f8f8d0) at /usr/src/sys= /amd64/amd64/trap.c:435 #15 0xffffffff808641b7 in calltrap () at /usr/src/sys/amd64/amd64/exception= .S:234 #16 0xffffffff81900c9a in dmu_objset_find_dp_impl (dcp=3D0xfffff80078cb0200= ) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_objset.c:1= 630 #17 0xffffffff81901189 in dmu_objset_find_dp_cb (arg=3D0xfffff80078cb0200) = at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_objset.c:1746 #18 0xffffffff818ab8d1 in taskq_run (arg=3D0xfffff800066d3d20, pending=3D1)= at /usr/src/sys/cddl/compat/opensolaris/kern/opensolaris_taskq.c:109 #19 0xffffffff805f3c8b in taskqueue_run_locked (queue=3D0xfffff800666b2500)= at /usr/src/sys/kern/subr_taskqueue.c:430 #20 0xffffffff805f4ad8 in taskqueue_thread_loop (arg=3D) at /usr/src/sys/kern/subr_taskqueue.c:683 #21 0xffffffff8055c77c in fork_exit (callout=3D0xffffffff805f4a00 , arg=3D0xfffff80060937470, frame=3D0xfffffe0094f8fc00) at /u= sr/src/sys/kern/kern_fork.c:1011 #22 0xffffffff808646ee in fork_trampoline () at /usr/src/sys/amd64/amd64/ex= ception.S:609 #23 0x0000000000000000 in ?? () Current language: auto; currently minimal (kgdb) f 16 #16 0xffffffff81900c9a in dmu_objset_find_dp_impl (dcp=3D0xfffff80078cb0200= ) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_objset.c:1= 630 1630 if (*dcp->dc_error !=3D 0) (kgdb) p *dcp $1 =3D {dc_tq =3D 0xfffff80060937470, dc_dp =3D 0xfffff8001c3d3000, dc_ddob= j =3D 4996, dc_func =3D 0xffffffff819a2320 , dc_arg =3D 0xfffff8= 0051c72200, dc_flags =3D 2, dc_error_lock =3D 0xfffffe0094653a48,=20 dc_error =3D 0xfffffe0094653a80} (kgdb) p *dcp->dc_error Cannot access memory at address 0xfffffe0094653a80 (kgdb) p *dcp->dc_error_lock Cannot access memory at address 0xfffffe0094653a48 (kgdb) p *dcp->dc_tq $2 =3D {tq_queue =3D 0xfffff800666b2500} (kgdb) p *dcp->dc_dp $3 =3D {dp_spa =3D 0xfffff800062eb000, dp_meta_objset =3D 0xfffff8003db0a40= 0, dp_root_dir =3D 0xfffff8000b0ba800, dp_mos_dir =3D 0xfffff8001cf05800, d= p_free_dir =3D 0xfffff8004ca79400, dp_leak_dir =3D 0x0,=20 dp_origin_snap =3D 0xfffff8001a1d8400, dp_root_dir_obj =3D 2, dp_vnrele_t= askq =3D 0xfffff800285be8e0, dp_meta_rootbp =3D {blk_dva =3D 0xfffff8001c3d= 3048, blk_prop =3D 9226475966770118659, blk_pad =3D 0xfffff8001c3d3080,=20 blk_phys_birth =3D 0, blk_birth =3D 27427, blk_fill =3D 5434, blk_cksum= =3D {zc_word =3D 0xfffff8001c3d30a8}}, dp_tmp_userrefs_obj =3D 0, dp_free_= bpobj =3D {bpo_lock =3D {lock_object =3D { lo_name =3D 0xffffffff81a58f27 "bpo->bpo_lock", lo_flags =3D 409600= 00, lo_data =3D 0, lo_witness =3D 0x0}, sx_lock =3D 1}, bpo_os =3D 0xfffff8= 003db0a400, bpo_object =3D 11, bpo_epb =3D 1024, bpo_havecomp =3D 1 '\001',= =20 bpo_havesubobj =3D 1 '\001', bpo_phys =3D 0xfffff8006a434e00, bpo_dbuf = =3D 0xfffff800385c76c0, bpo_cached_dbuf =3D 0x0}, dp_bptree_obj =3D 0, dp_e= mpty_bpobj =3D 45, dp_scan =3D 0xfffff8001c39dc00, dp_lock =3D { lock_object =3D {lo_name =3D 0xffffffff81a5f0a5 "dp->dp_lock", lo_flags= =3D 40960000, lo_data =3D 0, lo_witness =3D 0x0}, sx_lock =3D 1}, dp_space= avail_cv =3D { cv_description =3D 0xffffffff81a5f0b2 "dp->dp_spaceavail_cv", cv_waiter= s =3D 0}, dp_dirty_pertxg =3D 0xfffff8001c3d3168, dp_dirty_total =3D 0, dp_= mos_used_delta =3D 0, dp_mos_compressed_delta =3D 0,=20 dp_mos_uncompressed_delta =3D 0, dp_last_wakeup =3D 0, dp_tx =3D {tx_cpu = =3D 0xfffff8003db0a800, tx_sync_lock =3D {lock_object =3D {lo_name =3D 0xff= ffffff81a66a91 "tx->tx_sync_lock", lo_flags =3D 40960000, lo_data =3D 0,=20 lo_witness =3D 0x0}, sx_lock =3D 1}, tx_open_txg =3D 27428, tx_quie= sced_txg =3D 0, tx_syncing_txg =3D 0, tx_synced_txg =3D 0, tx_open_time =3D= 0, tx_sync_txg_waiting =3D 0, tx_quiesce_txg_waiting =3D 0,=20 tx_sync_more_cv =3D {cv_description =3D 0xffffffff81a66aa3 "tx->tx_sync= _more_cv", cv_waiters =3D 0}, tx_sync_done_cv =3D {cv_description =3D 0xfff= fffff81a66ab8 "tx->tx_sync_done_cv", cv_waiters =3D 0},=20 tx_quiesce_more_cv =3D {cv_description =3D 0xffffffff81a66acd "tx->tx_q= uiesce_more_cv", cv_waiters =3D 0}, tx_quiesce_done_cv =3D {cv_description = =3D 0xffffffff81a66ae5 "tx->tx_quiesce_done_cv", cv_waiters =3D 0},=20 tx_timeout_cv =3D {cv_description =3D 0x0, cv_waiters =3D 0}, tx_exit_c= v =3D {cv_description =3D 0xffffffff81a66afd "tx->tx_exit_cv", cv_waiters = =3D 0}, tx_threads =3D 0 '\0', tx_exiting =3D 0 '\0', tx_sync_thread =3D 0x= 0,=20 tx_quiesce_thread =3D 0x0, tx_commit_cb_taskq =3D 0x0}, dp_dirty_datase= ts =3D {tl_lock =3D {lock_object =3D {lo_name =3D 0xffffffff81a66b4b "tl->t= l_lock", lo_flags =3D 40960000, lo_data =3D 0, lo_witness =3D 0x0},=20 sx_lock =3D 1}, tl_offset =3D 376, tl_head =3D 0xfffff8001c3d32b8}, d= p_dirty_zilogs =3D {tl_lock =3D {lock_object =3D {lo_name =3D 0xffffffff81a= 66b4b "tl->tl_lock", lo_flags =3D 40960000, lo_data =3D 0,=20 lo_witness =3D 0x0}, sx_lock =3D 1}, tl_offset =3D 984, tl_head =3D= 0xfffff8001c3d3300}, dp_dirty_dirs =3D {tl_lock =3D {lock_object =3D {lo_n= ame =3D 0xffffffff81a66b4b "tl->tl_lock", lo_flags =3D 40960000,=20 lo_data =3D 0, lo_witness =3D 0x0}, sx_lock =3D 1}, tl_offset =3D 8= 0, tl_head =3D 0xfffff8001c3d3348}, dp_sync_tasks =3D {tl_lock =3D {lock_ob= ject =3D {lo_name =3D 0xffffffff81a66b4b "tl->tl_lock", lo_flags =3D 409600= 00,=20 lo_data =3D 0, lo_witness =3D 0x0}, sx_lock =3D 1}, tl_offset =3D 0= , tl_head =3D 0xfffff8001c3d3390}, dp_config_rwlock =3D {rr_lock =3D {lock_= object =3D {lo_name =3D 0xffffffff81a62115 "rrl->rr_lock",=20 lo_flags =3D 40960000, lo_data =3D 0, lo_witness =3D 0x0}, sx_lock = =3D 1}, rr_cv =3D {cv_description =3D 0xffffffff81a62123 "rrl->rr_cv", cv_w= aiters =3D 0}, rr_writer =3D 0x0, rr_anon_rcount =3D {rc_count =3D 0},=20 rr_linked_rcount =3D {rc_count =3D 3}, rr_writer_wanted =3D 0, rr_track= _all =3D 1}, dp_blkstats =3D 0x0} (kgdb) p *dcp->dc_dp->dp_spa $4 =3D {spa_name =3D 0xfffff800062eb000 "tor3", spa_comment =3D 0x0, spa_av= l =3D {avl_child =3D 0xfffff800062eb108, avl_pcb =3D 18446735277720248589},= spa_config =3D 0xfffff800608f4a20, spa_config_syncing =3D 0x0,=20 spa_config_splitting =3D 0x0, spa_load_info =3D 0xfffff800061d99c0, spa_c= onfig_txg =3D 27427, spa_sync_pass =3D 0, spa_state =3D POOL_STATE_ACTIVE, = spa_inject_ref =3D 0, spa_sync_on =3D 0 '\0',=20 spa_load_state =3D SPA_LOAD_OPEN, spa_import_flags =3D 0, spa_zio_taskq = =3D 0xfffff800062eb168, spa_dsl_pool =3D 0xfffff8001c3d3000, spa_is_initial= izing =3D 0, spa_normal_class =3D 0xfffff8001e504400,=20 spa_log_class =3D 0xfffff8004dd19000, spa_first_txg =3D 27428, spa_final_= txg =3D 18446744073709551615, spa_freeze_txg =3D 18446744073709551615, spa_= load_max_txg =3D 18446744073709551615, spa_claim_max_txg =3D 27428,=20 spa_loaded_ts =3D {tv_sec =3D 1450736623, tv_nsec =3D 578802346}, spa_met= a_objset =3D 0xfffff8003db0a400, spa_evicting_os_lock =3D {lock_object =3D = {lo_name =3D 0xffffffff81a660ce "spa->spa_evicting_os_lock",=20 lo_flags =3D 40960000, lo_data =3D 0, lo_witness =3D 0x0}, sx_lock = =3D 1}, spa_evicting_os_list =3D {list_size =3D 1008, list_offset =3D 256, = list_head =3D {list_next =3D 0xfffff800062eb378,=20 list_prev =3D 0xfffff800062eb378}}, spa_evicting_os_cv =3D {cv_descri= ption =3D 0xffffffff81a6619b "spa->spa_evicting_os_cv", cv_waiters =3D 0}, = spa_vdev_txg_list =3D {tl_lock =3D {lock_object =3D { lo_name =3D 0xffffffff81a66b4b "tl->tl_lock", lo_flags =3D 40960000= , lo_data =3D 0, lo_witness =3D 0x0}, sx_lock =3D 1}, tl_offset =3D 624, tl= _head =3D 0xfffff800062eb3c0}, spa_root_vdev =3D 0xfffff8006ab1a000,=20 spa_min_ashift =3D 12, spa_max_ashift =3D 12, spa_config_guid =3D 1327185= 4445444309143, spa_load_guid =3D 10536853844155556401, spa_last_synced_guid= =3D 0, spa_config_dirty_list =3D {list_size =3D 1704,=20 list_offset =3D 672, list_head =3D {list_next =3D 0xfffff800062eb418, l= ist_prev =3D 0xfffff800062eb418}}, spa_state_dirty_list =3D {list_size =3D = 1704, list_offset =3D 688, list_head =3D {list_next =3D 0xfffff800062eb438,= =20 list_prev =3D 0xfffff800062eb438}}, spa_spares =3D {sav_object =3D 0,= sav_config =3D 0x0, sav_vdevs =3D 0x0, sav_count =3D 0, sav_sync =3D 0, sa= v_pending =3D 0x0, sav_npending =3D 0}, spa_l2cache =3D {sav_object =3D 0,= =20 sav_config =3D 0x0, sav_vdevs =3D 0x0, sav_count =3D 0, sav_sync =3D 0,= sav_pending =3D 0x0, sav_npending =3D 0}, spa_label_features =3D 0xfffff80= 0061d99a0, spa_config_object =3D 27, spa_config_generation =3D 0,=20 spa_syncing_txg =3D 0, spa_deferred_bpobj =3D {bpo_lock =3D {lock_object = =3D {lo_name =3D 0xffffffff81a58f27 "bpo->bpo_lock", lo_flags =3D 40960000,= lo_data =3D 0, lo_witness =3D 0x0}, sx_lock =3D 1},=20 bpo_os =3D 0xfffff8003db0a400, bpo_object =3D 31, bpo_epb =3D 128, bpo_= havecomp =3D 1 '\001', bpo_havesubobj =3D 1 '\001', bpo_phys =3D 0xfffff800= 06330a00, bpo_dbuf =3D 0xfffff800385c75e8, bpo_cached_dbuf =3D 0x0},=20 spa_free_bplist =3D 0xfffff800062eb518, spa_cksum_salt =3D {zcs_bytes =3D= 0xfffff800062eb618 "=C3=BC=C3=AC`~-=C2=AAY"}, spa_cksum_tmpls_lock =3D {lo= ck_object =3D {lo_name =3D 0xffffffff81a66129 "spa->spa_cksum_tmpls_lock",= =20 lo_flags =3D 40960000, lo_data =3D 0, lo_witness =3D 0x0}, sx_lock = =3D 1}, spa_cksum_tmpls =3D 0xfffff800062eb658, spa_ubsync =3D {ub_magic = =3D 12235020, ub_version =3D 5000, ub_txg =3D 27427,=20 ub_guid_sum =3D 3109696218321734419, ub_timestamp =3D 1450604034, ub_ro= otbp =3D {blk_dva =3D 0xfffff800062eb6d8, blk_prop =3D 9226475966770118659,= blk_pad =3D 0xfffff800062eb710, blk_phys_birth =3D 0,=20 blk_birth =3D 27427, blk_fill =3D 5434, blk_cksum =3D {zc_word =3D 0x= fffff800062eb738}}, ub_software_version =3D 5000}, spa_uberblock =3D {ub_ma= gic =3D 12235020, ub_version =3D 5000, ub_txg =3D 27427,=20 ub_guid_sum =3D 3109696218321734419, ub_timestamp =3D 1450604034, ub_ro= otbp =3D {blk_dva =3D 0xfffff800062eb788, blk_prop =3D 9226475966770118659,= blk_pad =3D 0xfffff800062eb7c0, blk_phys_birth =3D 0,=20 blk_birth =3D 27427, blk_fill =3D 5434, blk_cksum =3D {zc_word =3D 0x= fffff800062eb7e8}}, ub_software_version =3D 5000}, spa_extreme_rewind =3D 0= , spa_last_io =3D 11912224, spa_scrub_lock =3D {lock_object =3D { lo_name =3D 0xffffffff81a66144 "spa->spa_scrub_lock", lo_flags =3D 40= 960000, lo_data =3D 0, lo_witness =3D 0x0}, sx_lock =3D 1}, spa_scrub_infli= ght =3D 0, spa_scrub_io_cv =3D { cv_description =3D 0xffffffff81a661c6 "spa->spa_scrub_io_cv", cv_waiter= s =3D 0}, spa_scrub_active =3D 0 '\0', spa_scrub_type =3D 0 '\0', spa_scrub= _finished =3D 0 '\0', spa_scrub_started =3D 0 '\0',=20 spa_scrub_reopen =3D 0 '\0', spa_scan_pass_start =3D 1450736643, spa_scan= _pass_exam =3D 0, spa_async_lock =3D {lock_object =3D {lo_name =3D 0xffffff= ff81a6608c "spa->spa_async_lock", lo_flags =3D 40960000, lo_data =3D 0,=20 lo_witness =3D 0x0}, sx_lock =3D 1}, spa_async_thread =3D 0x0, spa_as= ync_thread_vd =3D 0x0, spa_async_suspended =3D 0, spa_async_cv =3D {cv_desc= ription =3D 0xffffffff81a66188 "spa->spa_async_cv", cv_waiters =3D 0},=20 spa_async_tasks =3D 0, spa_root =3D 0x0, spa_ena =3D 0, spa_last_open_fai= led =3D 2, spa_last_ubsync_txg =3D 0, spa_last_ubsync_txg_ts =3D 0, spa_loa= d_txg =3D 27427, spa_load_txg_ts =3D 1450604034, spa_load_meta_errors =3D 0= ,=20 spa_load_data_errors =3D 0, spa_verify_min_txg =3D 27424, spa_errlog_lock= =3D {lock_object =3D {lo_name =3D 0xffffffff81a660b8 "spa->spa_errlog_lock= ", lo_flags =3D 40960000, lo_data =3D 0, lo_witness =3D 0x0},=20 sx_lock =3D 1}, spa_errlog_last =3D 0, spa_errlog_scrub =3D 0, spa_errl= ist_lock =3D {lock_object =3D {lo_name =3D 0xffffffff81a660a1 "spa->spa_err= list_lock", lo_flags =3D 40960000, lo_data =3D 0, lo_witness =3D 0x0},=20 sx_lock =3D 1}, spa_errlist_last =3D {avl_root =3D 0x0, avl_compar =3D = 0xffffffff8194ef20 , avl_offset =3D 40, avl_numnod= es =3D 0, avl_size =3D 64}, spa_errlist_scrub =3D {avl_root =3D 0x0,=20 avl_compar =3D 0xffffffff8194ef20 , avl_offset= =3D 40, avl_numnodes =3D 0, avl_size =3D 64}, spa_deflate =3D 1, spa_histo= ry =3D 32, spa_history_lock =3D {lock_object =3D { lo_name =3D 0xffffffff81a660e9 "spa->spa_history_lock", lo_flags =3D = 40960000, lo_data =3D 0, lo_witness =3D 0x0}, sx_lock =3D 1}, spa_pending_v= dev =3D 0x0, spa_props_lock =3D {lock_object =3D { lo_name =3D 0xffffffff81a66114 "spa->spa_props_lock", lo_flags =3D 40= 960000, lo_data =3D 0, lo_witness =3D 0x0}, sx_lock =3D 1}, spa_pool_props_= object =3D 33, spa_bootfs =3D 0, spa_failmode =3D 1, spa_delegation =3D 1,= =20 spa_config_list =3D {list_size =3D 24, list_offset =3D 0, list_head =3D {= list_next =3D 0xfffff800061d9a00, list_prev =3D 0xfffff800061d9a00}}, spa_a= sync_zio_root =3D 0xfffff80025dfeb50, spa_suspend_zio_root =3D 0x0,=20 spa_suspend_lock =3D {lock_object =3D {lo_name =3D 0xffffffff81a66159 "sp= a->spa_suspend_lock", lo_flags =3D 40960000, lo_data =3D 0, lo_witness =3D = 0x0}, sx_lock =3D 1}, spa_suspend_cv =3D { cv_description =3D 0xffffffff81a661dc "spa->spa_suspend_cv", cv_waiters= =3D 0}, spa_suspended =3D 0 '\0', spa_claiming =3D 1 '\001', spa_debug =3D= 0, spa_is_root =3D 0, spa_minref =3D 0, spa_mode =3D 3,=20 spa_log_state =3D SPA_LOG_UNKNOWN, spa_autoexpand =3D 0, spa_ddt =3D 0xff= fff800062ebaa8, spa_ddt_stat_object =3D 0, spa_dedup_ditto =3D 0, spa_dedup= _checksum =3D 8, spa_dspace =3D 10926396801024, spa_vdev_top_lock =3D { lock_object =3D {lo_name =3D 0xffffffff81a66170 "spa->spa_vdev_top_lock= ", lo_flags =3D 40960000, lo_data =3D 0, lo_witness =3D 0x0}, sx_lock =3D 1= }, spa_proc_lock =3D {lock_object =3D { lo_name =3D 0xffffffff81a66100 "spa->spa_proc_lock", lo_flags =3D 409= 60000, lo_data =3D 0, lo_witness =3D 0x0}, sx_lock =3D 1}, spa_proc_cv =3D = {cv_description =3D 0xffffffff81a661b4 "spa->spa_proc_cv",=20 cv_waiters =3D 0}, spa_proc_state =3D SPA_PROC_NONE, spa_proc =3D 0xfff= fffff81620028, spa_did =3D 0, spa_trim_thread =3D 0xfffff80056b074d0, spa_t= rim_lock =3D {lock_object =3D { lo_name =3D 0xffffffff81a6dfe1 "spa->spa_trim_lock", lo_flags =3D 409= 60000, lo_data =3D 0, lo_witness =3D 0x0}, sx_lock =3D 1}, spa_trim_cv =3D = {cv_description =3D 0xffffffff81a6dff5 "spa->spa_trim_cv",=20 cv_waiters =3D 131}, spa_autoreplace =3D 1, spa_vdev_locks =3D 0, spa_c= reation_version =3D 5000, spa_prev_software_version =3D 5000, spa_feat_for_= write_obj =3D 29, spa_feat_for_read_obj =3D 28, spa_feat_desc_obj =3D 30,=20 spa_feat_enabled_txg_obj =3D 34, spa_feat_refcount_cache =3D 0xfffff80006= 2ebbf8, spa_deadman_cycid =3D {c_links =3D {le =3D {le_next =3D 0x0, le_pre= v =3D 0x0}, sle =3D {sle_next =3D 0x0}, tqe =3D {tqe_next =3D 0x0,=20 tqe_prev =3D 0x0}}, c_time =3D 0, c_precision =3D 0, c_arg =3D 0x0,= c_func =3D 0, c_lock =3D 0x0, c_flags =3D 0, c_iflags =3D 16, c_cpu =3D 0}= , spa_deadman_calls =3D 0, spa_sync_starttime =3D 0,=20 spa_deadman_synctime =3D 1000000000000, spa_ccw_fail_time =3D 0, spa_conf= ig_lock =3D 0xfffff800062ebcb8, spa_refcount =3D {rc_count =3D 31}, spa_spl= itting_newspa =3D 0} tor3 is the imported ZFS pool. Given the location of the trap, this could be a regression caused by the import of illumos #5269 (zpool import slow) in r286686: https://svnweb.freebsd.org/base?view=3Drevision&revision=3Dr286686 Fabian --Sig_/I1kt9lqEZfT7evbjkYjI0DA Content-Type: application/pgp-signature Content-Description: OpenPGP digital signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iEYEARECAAYFAlZ5aEEACgkQBYqIVf93VJ3O9gCfbyiTf9e6mxcPBNYRBqASCDNJ 24oAoKcJY1XeR6qv5lytjzVb2mfZzlam =5+Dz -----END PGP SIGNATURE----- --Sig_/I1kt9lqEZfT7evbjkYjI0DA--