From owner-svn-src-all@freebsd.org Wed Oct 16 08:01:41 2019 Return-Path: Delivered-To: svn-src-all@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 8930915EAD1; Wed, 16 Oct 2019 08:01:41 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 46tPrj3vRqz4QHw; Wed, 16 Oct 2019 08:01:41 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 6A83F4083; Wed, 16 Oct 2019 08:01:41 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id x9G81fE3032526; Wed, 16 Oct 2019 08:01:41 GMT (envelope-from avg@FreeBSD.org) Received: (from avg@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id x9G81bLV032502; Wed, 16 Oct 2019 08:01:37 GMT (envelope-from avg@FreeBSD.org) Message-Id: <201910160801.x9G81bLV032502@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: avg set sender to avg@FreeBSD.org using -f From: Andriy Gapon Date: Wed, 16 Oct 2019 08:01:37 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r353630 - vendor-sys/illumos/dist/uts/common/fs/zfs vendor-sys/illumos/dist/uts/common/fs/zfs/sys vendor-sys/illumos/dist/uts/common/sys vendor/illumos/dist/lib/libzfs/common vendor/ill... X-SVN-Group: vendor-sys X-SVN-Commit-Author: avg X-SVN-Commit-Paths: vendor-sys/illumos/dist/uts/common/fs/zfs vendor-sys/illumos/dist/uts/common/fs/zfs/sys vendor-sys/illumos/dist/uts/common/sys vendor/illumos/dist/lib/libzfs/common vendor/illumos/dist/lib/libzpool/co... X-SVN-Commit-Revision: 353630 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-all@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "SVN commit messages for the entire src tree \(except for " user" and " projects" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Oct 2019 08:01:41 -0000 Author: avg Date: Wed Oct 16 08:01:36 2019 New Revision: 353630 URL: https://svnweb.freebsd.org/changeset/base/353630 Log: 10809 Performance optimization of AVL tree comparator functions illumos/illumos-gate@c4ab0d3f46036e85ad0700125c5a83cc139f55a3 https://github.com/illumos/illumos-gate/commit/c4ab0d3f46036e85ad0700125c5a83cc139f55a3 https://www.illumos.org/issues/10809 Port ZoL ee36c709c3d Performance optimization of AVL tree comparator functions From the ZoL commit msg: perf: 2.75x faster ddt_entry_compare() First 256bits of ddt_key_t is a block checksum, which are expected to be close to random data. Hence, on average, comparison only needs to look at first few bytes of the keys. To reduce number of conditional jump instructions, the result is computed as: sign(memcmp(k1, k2)). Sign of an integer 'a' can be obtained as: `(0 < a) - (a < 0)` := {-1, 0, 1}, which is computed efficiently. Synthetic performance evaluation of original and new algorithm over 1G random keys on 2.6GHz Intel(R) Xeon(R) CPU E5-2660 v3: old 6.85789 s new 2.49089 s perf: 2.8x faster vdev_queue_offset_compare() and vdev_queue_timestamp_compare() Compute the result directly instead of using conditionals perf: zfs_range_compare() Speedup between 1.1x - 2.5x, depending on compiler version and optimization level. perf: spa_error_entry_compare() `bcmp()` is not suitable for comparator use. Use `memcmp()` instead. perf: 2.8x faster metaslab_compare() and metaslab_rangesize_compare() perf: 2.8x faster zil_bp_compare() Portions contributed by: Jerry Jelinek Author: Gvozden Neskovic Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/ddt.c vendor-sys/illumos/dist/uts/common/fs/zfs/dnode.c vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_deadlist.c vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_deleg.c vendor-sys/illumos/dist/uts/common/fs/zfs/lz4.c vendor-sys/illumos/dist/uts/common/fs/zfs/metaslab.c vendor-sys/illumos/dist/uts/common/fs/zfs/range_tree.c vendor-sys/illumos/dist/uts/common/fs/zfs/sa.c vendor-sys/illumos/dist/uts/common/fs/zfs/spa.c vendor-sys/illumos/dist/uts/common/fs/zfs/spa_misc.c vendor-sys/illumos/dist/uts/common/fs/zfs/space_reftree.c vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zfs_context.h vendor-sys/illumos/dist/uts/common/fs/zfs/unique.c vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_cache.c vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_label.c vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_queue.c vendor-sys/illumos/dist/uts/common/fs/zfs/zap_micro.c vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_fuid.c vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_rlock.c vendor-sys/illumos/dist/uts/common/fs/zfs/zil.c vendor-sys/illumos/dist/uts/common/sys/avl.h Changes in other areas also in this revision: Modified: vendor/illumos/dist/lib/libzfs/common/libzfs_dataset.c vendor/illumos/dist/lib/libzfs/common/libzfs_iter.c vendor/illumos/dist/lib/libzfs/common/libzfs_sendrecv.c vendor/illumos/dist/lib/libzpool/common/sys/zfs_context.h Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/ddt.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/ddt.c Wed Oct 16 07:57:58 2019 (r353629) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/ddt.c Wed Oct 16 08:01:36 2019 (r353630) @@ -755,22 +755,31 @@ ddt_prefetch(spa_t *spa, const blkptr_t *bp) } } +/* + * Opaque struct used for ddt_key comparison + */ +#define DDT_KEY_CMP_LEN (sizeof (ddt_key_t) / sizeof (uint16_t)) + +typedef struct ddt_key_cmp { + uint16_t u16[DDT_KEY_CMP_LEN]; +} ddt_key_cmp_t; + int ddt_entry_compare(const void *x1, const void *x2) { const ddt_entry_t *dde1 = x1; const ddt_entry_t *dde2 = x2; - const uint64_t *u1 = (const uint64_t *)&dde1->dde_key; - const uint64_t *u2 = (const uint64_t *)&dde2->dde_key; + const ddt_key_cmp_t *k1 = (const ddt_key_cmp_t *)&dde1->dde_key; + const ddt_key_cmp_t *k2 = (const ddt_key_cmp_t *)&dde2->dde_key; + int32_t cmp = 0; - for (int i = 0; i < DDT_KEY_WORDS; i++) { - if (u1[i] < u2[i]) - return (-1); - if (u1[i] > u2[i]) - return (1); + for (int i = 0; i < DDT_KEY_CMP_LEN; i++) { + cmp = (int32_t)k1->u16[i] - (int32_t)k2->u16[i]; + if (likely(cmp)) + break; } - return (0); + return (AVL_ISIGN(cmp)); } static ddt_t * Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/dnode.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/dnode.c Wed Oct 16 07:57:58 2019 (r353629) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/dnode.c Wed Oct 16 08:01:36 2019 (r353630) @@ -90,19 +90,13 @@ dbuf_compare(const void *x1, const void *x2) const dmu_buf_impl_t *d1 = x1; const dmu_buf_impl_t *d2 = x2; - if (d1->db_level < d2->db_level) { - return (-1); - } - if (d1->db_level > d2->db_level) { - return (1); - } + int cmp = AVL_CMP(d1->db_level, d2->db_level); + if (likely(cmp)) + return (cmp); - if (d1->db_blkid < d2->db_blkid) { - return (-1); - } - if (d1->db_blkid > d2->db_blkid) { - return (1); - } + cmp = AVL_CMP(d1->db_blkid, d2->db_blkid); + if (likely(cmp)) + return (cmp); if (d1->db_state == DB_SEARCH) { ASSERT3S(d2->db_state, !=, DB_SEARCH); @@ -112,13 +106,7 @@ dbuf_compare(const void *x1, const void *x2) return (1); } - if ((uintptr_t)d1 < (uintptr_t)d2) { - return (-1); - } - if ((uintptr_t)d1 > (uintptr_t)d2) { - return (1); - } - return (0); + return (AVL_PCMP(d1, d2)); } /* ARGSUSED */ Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_deadlist.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_deadlist.c Wed Oct 16 07:57:58 2019 (r353629) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_deadlist.c Wed Oct 16 08:01:36 2019 (r353630) @@ -55,15 +55,10 @@ static int dsl_deadlist_compare(const void *arg1, const void *arg2) { - const dsl_deadlist_entry_t *dle1 = arg1; - const dsl_deadlist_entry_t *dle2 = arg2; + const dsl_deadlist_entry_t *dle1 = (const dsl_deadlist_entry_t *)arg1; + const dsl_deadlist_entry_t *dle2 = (const dsl_deadlist_entry_t *)arg2; - if (dle1->dle_mintxg < dle2->dle_mintxg) - return (-1); - else if (dle1->dle_mintxg > dle2->dle_mintxg) - return (+1); - else - return (0); + return (AVL_CMP(dle1->dle_mintxg, dle2->dle_mintxg)); } static void Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_deleg.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_deleg.c Wed Oct 16 07:57:58 2019 (r353629) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_deleg.c Wed Oct 16 08:01:36 2019 (r353630) @@ -384,14 +384,13 @@ typedef struct perm_set { static int perm_set_compare(const void *arg1, const void *arg2) { - const perm_set_t *node1 = arg1; - const perm_set_t *node2 = arg2; + const perm_set_t *node1 = (const perm_set_t *)arg1; + const perm_set_t *node2 = (const perm_set_t *)arg2; int val; val = strcmp(node1->p_setname, node2->p_setname); - if (val == 0) - return (0); - return (val > 0 ? 1 : -1); + + return (AVL_ISIGN(val)); } /* Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/lz4.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/lz4.c Wed Oct 16 07:57:58 2019 (r353629) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/lz4.c Wed Oct 16 08:01:36 2019 (r353630) @@ -284,8 +284,13 @@ lz4_decompress(void *s_start, void *d_start, size_t s_ #define expect(expr, value) (expr) #endif +#ifndef likely #define likely(expr) expect((expr) != 0, 1) +#endif + +#ifndef unlikely #define unlikely(expr) expect((expr) != 0, 0) +#endif /* Basic types */ #if defined(_MSC_VER) Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/metaslab.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/metaslab.c Wed Oct 16 07:57:58 2019 (r353629) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/metaslab.c Wed Oct 16 08:01:36 2019 (r353630) @@ -451,8 +451,8 @@ metaslab_class_expandable_space(metaslab_class_t *mc) static int metaslab_compare(const void *x1, const void *x2) { - const metaslab_t *m1 = x1; - const metaslab_t *m2 = x2; + const metaslab_t *m1 = (const metaslab_t *)x1; + const metaslab_t *m2 = (const metaslab_t *)x2; int sort1 = 0; int sort2 = 0; @@ -478,22 +478,13 @@ metaslab_compare(const void *x1, const void *x2) if (sort1 > sort2) return (1); - if (m1->ms_weight < m2->ms_weight) - return (1); - if (m1->ms_weight > m2->ms_weight) - return (-1); + int cmp = AVL_CMP(m2->ms_weight, m1->ms_weight); + if (likely(cmp)) + return (cmp); - /* - * If the weights are identical, use the offset to force uniqueness. - */ - if (m1->ms_start < m2->ms_start) - return (-1); - if (m1->ms_start > m2->ms_start) - return (1); + IMPLY(AVL_CMP(m1->ms_start, m2->ms_start) == 0, m1 == m2); - ASSERT3P(m1, ==, m2); - - return (0); + return (AVL_CMP(m1->ms_start, m2->ms_start)); } /* @@ -1104,18 +1095,11 @@ metaslab_rangesize_compare(const void *x1, const void uint64_t rs_size1 = r1->rs_end - r1->rs_start; uint64_t rs_size2 = r2->rs_end - r2->rs_start; - if (rs_size1 < rs_size2) - return (-1); - if (rs_size1 > rs_size2) - return (1); + int cmp = AVL_CMP(rs_size1, rs_size2); + if (likely(cmp)) + return (cmp); - if (r1->rs_start < r2->rs_start) - return (-1); - - if (r1->rs_start > r2->rs_start) - return (1); - - return (0); + return (AVL_CMP(r1->rs_start, r2->rs_start)); } /* Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/range_tree.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/range_tree.c Wed Oct 16 07:57:58 2019 (r353629) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/range_tree.c Wed Oct 16 08:01:36 2019 (r353630) @@ -109,20 +109,13 @@ range_tree_stat_decr(range_tree_t *rt, range_seg_t *rs static int range_tree_seg_compare(const void *x1, const void *x2) { - const range_seg_t *r1 = x1; - const range_seg_t *r2 = x2; + const range_seg_t *r1 = (const range_seg_t *)x1; + const range_seg_t *r2 = (const range_seg_t *)x2; - if (r1->rs_start < r2->rs_start) { - if (r1->rs_end > r2->rs_start) - return (0); - return (-1); - } - if (r1->rs_start > r2->rs_start) { - if (r1->rs_start < r2->rs_end) - return (0); - return (1); - } - return (0); + ASSERT3U(r1->rs_start, <=, r1->rs_end); + ASSERT3U(r2->rs_start, <=, r2->rs_end); + + return ((r1->rs_start >= r2->rs_end) - (r1->rs_end <= r2->rs_start)); } range_tree_t * Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/sa.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/sa.c Wed Oct 16 07:57:58 2019 (r353629) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/sa.c Wed Oct 16 08:01:36 2019 (r353630) @@ -243,31 +243,23 @@ sa_cache_fini(void) static int layout_num_compare(const void *arg1, const void *arg2) { - const sa_lot_t *node1 = arg1; - const sa_lot_t *node2 = arg2; + const sa_lot_t *node1 = (const sa_lot_t *)arg1; + const sa_lot_t *node2 = (const sa_lot_t *)arg2; - if (node1->lot_num > node2->lot_num) - return (1); - else if (node1->lot_num < node2->lot_num) - return (-1); - return (0); + return (AVL_CMP(node1->lot_num, node2->lot_num)); } static int layout_hash_compare(const void *arg1, const void *arg2) { - const sa_lot_t *node1 = arg1; - const sa_lot_t *node2 = arg2; + const sa_lot_t *node1 = (const sa_lot_t *)arg1; + const sa_lot_t *node2 = (const sa_lot_t *)arg2; - if (node1->lot_hash > node2->lot_hash) - return (1); - if (node1->lot_hash < node2->lot_hash) - return (-1); - if (node1->lot_instance > node2->lot_instance) - return (1); - if (node1->lot_instance < node2->lot_instance) - return (-1); - return (0); + int cmp = AVL_CMP(node1->lot_hash, node2->lot_hash); + if (likely(cmp)) + return (cmp); + + return (AVL_CMP(node1->lot_instance, node2->lot_instance)); } boolean_t Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/spa.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/spa.c Wed Oct 16 07:57:58 2019 (r353629) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/spa.c Wed Oct 16 08:01:36 2019 (r353630) @@ -882,19 +882,14 @@ spa_change_guid(spa_t *spa) static int spa_error_entry_compare(const void *a, const void *b) { - spa_error_entry_t *sa = (spa_error_entry_t *)a; - spa_error_entry_t *sb = (spa_error_entry_t *)b; + const spa_error_entry_t *sa = (const spa_error_entry_t *)a; + const spa_error_entry_t *sb = (const spa_error_entry_t *)b; int ret; - ret = bcmp(&sa->se_bookmark, &sb->se_bookmark, + ret = memcmp(&sa->se_bookmark, &sb->se_bookmark, sizeof (zbookmark_phys_t)); - if (ret < 0) - return (-1); - else if (ret > 0) - return (1); - else - return (0); + return (AVL_ISIGN(ret)); } /* Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/spa_misc.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/spa_misc.c Wed Oct 16 07:57:58 2019 (r353629) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/spa_misc.c Wed Oct 16 08:01:36 2019 (r353630) @@ -897,18 +897,13 @@ typedef struct spa_aux { int aux_count; } spa_aux_t; -static int +static inline int spa_aux_compare(const void *a, const void *b) { - const spa_aux_t *sa = a; - const spa_aux_t *sb = b; + const spa_aux_t *sa = (const spa_aux_t *)a; + const spa_aux_t *sb = (const spa_aux_t *)b; - if (sa->aux_guid < sb->aux_guid) - return (-1); - else if (sa->aux_guid > sb->aux_guid) - return (1); - else - return (0); + return (AVL_CMP(sa->aux_guid, sb->aux_guid)); } void @@ -1878,11 +1873,8 @@ spa_name_compare(const void *a1, const void *a2) int s; s = strcmp(s1->spa_name, s2->spa_name); - if (s > 0) - return (1); - if (s < 0) - return (-1); - return (0); + + return (AVL_ISIGN(s)); } int Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/space_reftree.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/space_reftree.c Wed Oct 16 07:57:58 2019 (r353629) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/space_reftree.c Wed Oct 16 08:01:36 2019 (r353630) @@ -54,20 +54,14 @@ static int space_reftree_compare(const void *x1, const void *x2) { - const space_ref_t *sr1 = x1; - const space_ref_t *sr2 = x2; + const space_ref_t *sr1 = (const space_ref_t *)x1; + const space_ref_t *sr2 = (const space_ref_t *)x2; - if (sr1->sr_offset < sr2->sr_offset) - return (-1); - if (sr1->sr_offset > sr2->sr_offset) - return (1); + int cmp = AVL_CMP(sr1->sr_offset, sr2->sr_offset); + if (likely(cmp)) + return (cmp); - if (sr1 < sr2) - return (-1); - if (sr1 > sr2) - return (1); - - return (0); + return (AVL_PCMP(sr1, sr2)); } void Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zfs_context.h ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zfs_context.h Wed Oct 16 07:57:58 2019 (r353629) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zfs_context.h Wed Oct 16 08:01:36 2019 (r353630) @@ -73,6 +73,14 @@ extern "C" { #include #include +#if (GCC_VERSION >= 302) || (__INTEL_COMPILER >= 800) || defined(__clang__) +#define _zfs_expect(expr, value) (__builtin_expect((expr), (value))) +#else +#define _zfs_expect(expr, value) (expr) +#endif + +#define likely(x) _zfs_expect((x) != 0, 1) + #define CPU_SEQID (CPU->cpu_seqid) #ifdef __cplusplus Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/unique.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/unique.c Wed Oct 16 07:57:58 2019 (r353629) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/unique.c Wed Oct 16 08:01:36 2019 (r353630) @@ -42,14 +42,10 @@ typedef struct unique { static int unique_compare(const void *a, const void *b) { - const unique_t *una = a; - const unique_t *unb = b; + const unique_t *una = (const unique_t *)a; + const unique_t *unb = (const unique_t *)b; - if (una->un_value < unb->un_value) - return (-1); - if (una->un_value > unb->un_value) - return (+1); - return (0); + return (AVL_CMP(una->un_value, unb->un_value)); } void Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_cache.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_cache.c Wed Oct 16 07:57:58 2019 (r353629) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_cache.c Wed Oct 16 08:01:36 2019 (r353630) @@ -105,29 +105,24 @@ static vdc_stats_t vdc_stats = { #define VDCSTAT_BUMP(stat) atomic_inc_64(&vdc_stats.stat.value.ui64); -static int +static inline int vdev_cache_offset_compare(const void *a1, const void *a2) { - const vdev_cache_entry_t *ve1 = a1; - const vdev_cache_entry_t *ve2 = a2; + const vdev_cache_entry_t *ve1 = (const vdev_cache_entry_t *)a1; + const vdev_cache_entry_t *ve2 = (const vdev_cache_entry_t *)a2; - if (ve1->ve_offset < ve2->ve_offset) - return (-1); - if (ve1->ve_offset > ve2->ve_offset) - return (1); - return (0); + return (AVL_CMP(ve1->ve_offset, ve2->ve_offset)); } static int vdev_cache_lastused_compare(const void *a1, const void *a2) { - const vdev_cache_entry_t *ve1 = a1; - const vdev_cache_entry_t *ve2 = a2; + const vdev_cache_entry_t *ve1 = (const vdev_cache_entry_t *)a1; + const vdev_cache_entry_t *ve2 = (const vdev_cache_entry_t *)a2; - if (ve1->ve_lastused < ve2->ve_lastused) - return (-1); - if (ve1->ve_lastused > ve2->ve_lastused) - return (1); + int cmp = AVL_CMP(ve1->ve_lastused, ve2->ve_lastused); + if (likely(cmp)) + return (cmp); /* * Among equally old entries, sort by offset to ensure uniqueness. Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_label.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_label.c Wed Oct 16 07:57:58 2019 (r353629) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_label.c Wed Oct 16 08:01:36 2019 (r353630) @@ -989,19 +989,13 @@ retry: * among uberblocks with equal txg, choose the one with the latest timestamp. */ static int -vdev_uberblock_compare(uberblock_t *ub1, uberblock_t *ub2) +vdev_uberblock_compare(const uberblock_t *ub1, const uberblock_t *ub2) { - if (ub1->ub_txg < ub2->ub_txg) - return (-1); - if (ub1->ub_txg > ub2->ub_txg) - return (1); + int cmp = AVL_CMP(ub1->ub_txg, ub2->ub_txg); + if (likely(cmp)) + return (cmp); - if (ub1->ub_timestamp < ub2->ub_timestamp) - return (-1); - if (ub1->ub_timestamp > ub2->ub_timestamp) - return (1); - - return (0); + return (AVL_CMP(ub1->ub_timestamp, ub2->ub_timestamp)); } struct ubl_cbdata { Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_queue.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_queue.c Wed Oct 16 07:57:58 2019 (r353629) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_queue.c Wed Oct 16 08:01:36 2019 (r353630) @@ -202,20 +202,15 @@ int zfs_vdev_def_queue_depth = 32; int vdev_queue_offset_compare(const void *x1, const void *x2) { - const zio_t *z1 = x1; - const zio_t *z2 = x2; + const zio_t *z1 = (const zio_t *)x1; + const zio_t *z2 = (const zio_t *)x2; - if (z1->io_offset < z2->io_offset) - return (-1); - if (z1->io_offset > z2->io_offset) - return (1); + int cmp = AVL_CMP(z1->io_offset, z2->io_offset); - if (z1 < z2) - return (-1); - if (z1 > z2) - return (1); + if (likely(cmp)) + return (cmp); - return (0); + return (AVL_PCMP(z1, z2)); } static inline avl_tree_t * @@ -237,20 +232,15 @@ vdev_queue_type_tree(vdev_queue_t *vq, zio_type_t t) int vdev_queue_timestamp_compare(const void *x1, const void *x2) { - const zio_t *z1 = x1; - const zio_t *z2 = x2; + const zio_t *z1 = (const zio_t *)x1; + const zio_t *z2 = (const zio_t *)x2; - if (z1->io_timestamp < z2->io_timestamp) - return (-1); - if (z1->io_timestamp > z2->io_timestamp) - return (1); + int cmp = AVL_CMP(z1->io_timestamp, z2->io_timestamp); - if (z1 < z2) - return (-1); - if (z1 > z2) - return (1); + if (likely(cmp)) + return (cmp); - return (0); + return (AVL_PCMP(z1, z2)); } void Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/zap_micro.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/zap_micro.c Wed Oct 16 07:57:58 2019 (r353629) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/zap_micro.c Wed Oct 16 08:01:36 2019 (r353630) @@ -281,15 +281,11 @@ mze_compare(const void *arg1, const void *arg2) const mzap_ent_t *mze1 = arg1; const mzap_ent_t *mze2 = arg2; - if (mze1->mze_hash > mze2->mze_hash) - return (+1); - if (mze1->mze_hash < mze2->mze_hash) - return (-1); - if (mze1->mze_cd > mze2->mze_cd) - return (+1); - if (mze1->mze_cd < mze2->mze_cd) - return (-1); - return (0); + int cmp = AVL_CMP(mze1->mze_hash, mze2->mze_hash); + if (likely(cmp)) + return (cmp); + + return (AVL_CMP(mze1->mze_cd, mze2->mze_cd)); } static void Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_fuid.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_fuid.c Wed Oct 16 07:57:58 2019 (r353629) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_fuid.c Wed Oct 16 08:01:36 2019 (r353630) @@ -71,14 +71,10 @@ static char *nulldomain = ""; static int idx_compare(const void *arg1, const void *arg2) { - const fuid_domain_t *node1 = arg1; - const fuid_domain_t *node2 = arg2; + const fuid_domain_t *node1 = (const fuid_domain_t *)arg1; + const fuid_domain_t *node2 = (const fuid_domain_t *)arg2; - if (node1->f_idx < node2->f_idx) - return (-1); - else if (node1->f_idx > node2->f_idx) - return (1); - return (0); + return (AVL_CMP(node1->f_idx, node2->f_idx)); } /* @@ -87,14 +83,13 @@ idx_compare(const void *arg1, const void *arg2) static int domain_compare(const void *arg1, const void *arg2) { - const fuid_domain_t *node1 = arg1; - const fuid_domain_t *node2 = arg2; + const fuid_domain_t *node1 = (const fuid_domain_t *)arg1; + const fuid_domain_t *node2 = (const fuid_domain_t *)arg2; int val; val = strcmp(node1->f_ksid->kd_name, node2->f_ksid->kd_name); - if (val == 0) - return (0); - return (val > 0 ? 1 : -1); + + return (AVL_ISIGN(val)); } void Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_rlock.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_rlock.c Wed Oct 16 07:57:58 2019 (r353629) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_rlock.c Wed Oct 16 08:01:36 2019 (r353630) @@ -106,14 +106,10 @@ static int rangelock_compare(const void *arg1, const void *arg2) { - const locked_range_t *rl1 = arg1; - const locked_range_t *rl2 = arg2; + const locked_range_t *rl1 = (const locked_range_t *)arg1; + const locked_range_t *rl2 = (const locked_range_t *)arg2; - if (rl1->lr_offset > rl2->lr_offset) - return (1); - if (rl1->lr_offset < rl2->lr_offset) - return (-1); - return (0); + return (AVL_CMP(rl1->lr_offset, rl2->lr_offset)); } /* Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/zil.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/zil.c Wed Oct 16 07:57:58 2019 (r353629) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/zil.c Wed Oct 16 08:01:36 2019 (r353630) @@ -123,17 +123,11 @@ zil_bp_compare(const void *x1, const void *x2) const dva_t *dva1 = &((zil_bp_node_t *)x1)->zn_dva; const dva_t *dva2 = &((zil_bp_node_t *)x2)->zn_dva; - if (DVA_GET_VDEV(dva1) < DVA_GET_VDEV(dva2)) - return (-1); - if (DVA_GET_VDEV(dva1) > DVA_GET_VDEV(dva2)) - return (1); + int cmp = AVL_CMP(DVA_GET_VDEV(dva1), DVA_GET_VDEV(dva2)); + if (likely(cmp)) + return (cmp); - if (DVA_GET_OFFSET(dva1) < DVA_GET_OFFSET(dva2)) - return (-1); - if (DVA_GET_OFFSET(dva1) > DVA_GET_OFFSET(dva2)) - return (1); - - return (0); + return (AVL_CMP(DVA_GET_OFFSET(dva1), DVA_GET_OFFSET(dva2))); } static void @@ -495,12 +489,7 @@ zil_lwb_vdev_compare(const void *x1, const void *x2) const uint64_t v1 = ((zil_vdev_node_t *)x1)->zv_vdev; const uint64_t v2 = ((zil_vdev_node_t *)x2)->zv_vdev; - if (v1 < v2) - return (-1); - if (v1 > v2) - return (1); - - return (0); + return (AVL_CMP(v1, v2)); } static lwb_t * @@ -1729,12 +1718,7 @@ zil_aitx_compare(const void *x1, const void *x2) const uint64_t o1 = ((itx_async_node_t *)x1)->ia_foid; const uint64_t o2 = ((itx_async_node_t *)x2)->ia_foid; - if (o1 < o2) - return (-1); - if (o1 > o2) - return (1); - - return (0); + return (AVL_CMP(o1, o2)); } /* Modified: vendor-sys/illumos/dist/uts/common/sys/avl.h ============================================================================== --- vendor-sys/illumos/dist/uts/common/sys/avl.h Wed Oct 16 07:57:58 2019 (r353629) +++ vendor-sys/illumos/dist/uts/common/sys/avl.h Wed Oct 16 08:01:36 2019 (r353630) @@ -105,6 +105,13 @@ extern "C" { * as is needed for any linked list implementation. */ +/* + * AVL comparator helpers + */ +#define AVL_ISIGN(a) (((a) > 0) - ((a) < 0)) +#define AVL_CMP(a, b) (((a) > (b)) - ((a) < (b))) +#define AVL_PCMP(a, b) \ + (((uintptr_t)(a) > (uintptr_t)(b)) - ((uintptr_t)(a) < (uintptr_t)(b))) /* * Type used for the root of the AVL tree.