From owner-svn-src-stable-12@freebsd.org Thu Feb 20 01:03:45 2020 Return-Path: Delivered-To: svn-src-stable-12@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 24EF324A1F1; Thu, 20 Feb 2020 01:03:45 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 48NGYr6tyfz4L1S; Thu, 20 Feb 2020 01:03:44 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id E5B0319A01; Thu, 20 Feb 2020 01:03:44 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id 01K13i9x073982; Thu, 20 Feb 2020 01:03:44 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id 01K13i0a073980; Thu, 20 Feb 2020 01:03:44 GMT (envelope-from mav@FreeBSD.org) Message-Id: <202002200103.01K13i0a073980@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Thu, 20 Feb 2020 01:03:44 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-12@freebsd.org Subject: svn commit: r358137 - in stable/12/sys/cddl/contrib/opensolaris/uts/common/fs/zfs: . sys X-SVN-Group: stable-12 X-SVN-Commit-Author: mav X-SVN-Commit-Paths: in stable/12/sys/cddl/contrib/opensolaris/uts/common/fs/zfs: . sys X-SVN-Commit-Revision: 358137 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable-12@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: SVN commit messages for only the 12-stable src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 20 Feb 2020 01:03:45 -0000 Author: mav Date: Thu Feb 20 01:03:44 2020 New Revision: 358137 URL: https://svnweb.freebsd.org/changeset/base/358137 Log: MFC r357639: Reduce number of atomic_add() calls in aggsum. Previous code used 4 atomics to do aggsum_flush_bucket() and 2 more to re-borrow after the flush. But since asc_borrowed and asc_delta are accessed only while holding asc_lock, it makes no any sense to modify as_lower_bound and as_upper_bound in multiple steps. Instead of that the new code uses only 2 atomics in all the cases, one per as_*_bound variable. I think even that is overkill, simple atomic store and load could be used here, since all modifications are done under the as_lock, but there are no such primitives in ZFS code now. While there, make borrow code consider previous borrow value, so that on mixed request patterns reduce chance of needing to borrow again if much larger request follows tiny one that needed borrow. Also reduce as_numbuckets from uint64_t to u_int. It makes no sense to use so large division operation on every aggsum_add(). Modified: stable/12/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/aggsum.c stable/12/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/aggsum.h Directory Properties: stable/12/ (props changed) Modified: stable/12/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/aggsum.c ============================================================================== --- stable/12/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/aggsum.c Thu Feb 20 00:46:22 2020 (r358136) +++ stable/12/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/aggsum.c Thu Feb 20 01:03:44 2020 (r358137) @@ -125,13 +125,11 @@ aggsum_flush_bucket(aggsum_t *as, struct aggsum_bucket * We use atomic instructions for this because we read the upper and * lower bounds without the lock, so we need stores to be atomic. */ - atomic_add_64((volatile uint64_t *)&as->as_lower_bound, asb->asc_delta); - atomic_add_64((volatile uint64_t *)&as->as_upper_bound, asb->asc_delta); - asb->asc_delta = 0; - atomic_add_64((volatile uint64_t *)&as->as_upper_bound, - -asb->asc_borrowed); atomic_add_64((volatile uint64_t *)&as->as_lower_bound, - asb->asc_borrowed); + asb->asc_delta + asb->asc_borrowed); + atomic_add_64((volatile uint64_t *)&as->as_upper_bound, + asb->asc_delta - asb->asc_borrowed); + asb->asc_delta = 0; asb->asc_borrowed = 0; } @@ -163,40 +161,43 @@ aggsum_value(aggsum_t *as) return (rv); } -static void -aggsum_borrow(aggsum_t *as, int64_t delta, struct aggsum_bucket *asb) -{ - int64_t abs_delta = (delta < 0 ? -delta : delta); - mutex_enter(&as->as_lock); - mutex_enter(&asb->asc_lock); - - aggsum_flush_bucket(as, asb); - - atomic_add_64((volatile uint64_t *)&as->as_upper_bound, abs_delta); - atomic_add_64((volatile uint64_t *)&as->as_lower_bound, -abs_delta); - asb->asc_borrowed = abs_delta; - - mutex_exit(&asb->asc_lock); - mutex_exit(&as->as_lock); -} - void aggsum_add(aggsum_t *as, int64_t delta) { struct aggsum_bucket *asb = &as->as_buckets[CPU_SEQID % as->as_numbuckets]; + int64_t borrow; - for (;;) { - mutex_enter(&asb->asc_lock); - if (asb->asc_delta + delta <= (int64_t)asb->asc_borrowed && - asb->asc_delta + delta >= -(int64_t)asb->asc_borrowed) { - asb->asc_delta += delta; - mutex_exit(&asb->asc_lock); - return; - } + /* Try fast path if we already borrowed enough before. */ + mutex_enter(&asb->asc_lock); + if (asb->asc_delta + delta <= (int64_t)asb->asc_borrowed && + asb->asc_delta + delta >= -(int64_t)asb->asc_borrowed) { + asb->asc_delta += delta; mutex_exit(&asb->asc_lock); - aggsum_borrow(as, delta * aggsum_borrow_multiplier, asb); + return; } + mutex_exit(&asb->asc_lock); + + /* + * We haven't borrowed enough. Take the global lock and borrow + * considering what is requested now and what we borrowed before. + */ + borrow = (delta < 0 ? -delta : delta) * aggsum_borrow_multiplier; + mutex_enter(&as->as_lock); + mutex_enter(&asb->asc_lock); + delta += asb->asc_delta; + asb->asc_delta = 0; + if (borrow >= asb->asc_borrowed) + borrow -= asb->asc_borrowed; + else + borrow = (borrow - (int64_t)asb->asc_borrowed) / 4; + asb->asc_borrowed += borrow; + atomic_add_64((volatile uint64_t *)&as->as_lower_bound, + delta - borrow); + atomic_add_64((volatile uint64_t *)&as->as_upper_bound, + delta + borrow); + mutex_exit(&asb->asc_lock); + mutex_exit(&as->as_lock); } /* Modified: stable/12/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/aggsum.h ============================================================================== --- stable/12/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/aggsum.h Thu Feb 20 00:46:22 2020 (r358136) +++ stable/12/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/aggsum.h Thu Feb 20 01:03:44 2020 (r358137) @@ -39,7 +39,7 @@ typedef struct aggsum { kmutex_t as_lock; int64_t as_lower_bound; int64_t as_upper_bound; - uint64_t as_numbuckets; + uint_t as_numbuckets; aggsum_bucket_t *as_buckets; } aggsum_t;