From owner-svn-src-head@freebsd.org Thu Dec 27 19:15:25 2018 Return-Path: Delivered-To: svn-src-head@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 37CF714207F8; Thu, 27 Dec 2018 19:15:25 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D187088302; Thu, 27 Dec 2018 19:15:24 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id C4CA8FF6A; Thu, 27 Dec 2018 19:15:24 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id wBRJFOxx080132; Thu, 27 Dec 2018 19:15:24 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id wBRJFObc080131; Thu, 27 Dec 2018 19:15:24 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201812271915.wBRJFObc080131@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Thu, 27 Dec 2018 19:15:24 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org Subject: svn commit: r342558 - head/sys/geom X-SVN-Group: head X-SVN-Commit-Author: mav X-SVN-Commit-Paths: head/sys/geom X-SVN-Commit-Revision: 342558 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: D187088302 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-2.95 / 15.00]; local_wl_from(0.00)[FreeBSD.org]; NEURAL_HAM_MEDIUM(-1.00)[-0.999,0]; NEURAL_HAM_SHORT(-0.95)[-0.950,0]; ASN(0.00)[asn:11403, ipnet:2610:1c1:1::/48, country:US]; NEURAL_HAM_LONG(-1.00)[-0.999,0] X-BeenThere: svn-src-head@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: SVN commit messages for the src tree for head/-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Dec 2018 19:15:25 -0000 Author: mav Date: Thu Dec 27 19:15:24 2018 New Revision: 342558 URL: https://svnweb.freebsd.org/changeset/base/342558 Log: Switch from mutexes to atomics in GEOM_DEV I/O path. Mutexes in I/O path there were used twice per I/O to atomically access several variables to close and/or destroy the device on last request completion. I found the way to fit all required info into one integer, suitable for atomic operations. It opened race window on device close, but addition of timeout to the msleep() there should cover it. Profiling shows removal of significant spinning time on those mutexes and IOPS increase from ~600K to >800K to NVMe on 72-core systems. MFC after: 1 month Sponsored by: iXsystems, Inc. Modified: head/sys/geom/geom_dev.c Modified: head/sys/geom/geom_dev.c ============================================================================== --- head/sys/geom/geom_dev.c Thu Dec 27 18:28:19 2018 (r342557) +++ head/sys/geom/geom_dev.c Thu Dec 27 19:15:24 2018 (r342558) @@ -64,7 +64,10 @@ struct g_dev_softc { struct cdev *sc_dev; struct cdev *sc_alias; int sc_open; - int sc_active; + u_int sc_active; +#define SC_A_DESTROY (1 << 31) +#define SC_A_OPEN (1 << 30) +#define SC_A_ACTIVE (SC_A_OPEN - 1) }; static d_open_t g_dev_open; @@ -420,9 +423,13 @@ g_dev_open(struct cdev *dev, int flags, int fmt, struc if (error == 0) { sc = cp->private; mtx_lock(&sc->sc_mtx); - if (sc->sc_open == 0 && sc->sc_active != 0) + if (sc->sc_open == 0 && (sc->sc_active & SC_A_ACTIVE) != 0) wakeup(&sc->sc_active); sc->sc_open += r + w + e; + if (sc->sc_open == 0) + atomic_clear_int(&sc->sc_active, SC_A_OPEN); + else + atomic_set_int(&sc->sc_active, SC_A_OPEN); mtx_unlock(&sc->sc_mtx); } return (error); @@ -465,8 +472,12 @@ g_dev_close(struct cdev *dev, int flags, int fmt, stru sc = cp->private; mtx_lock(&sc->sc_mtx); sc->sc_open += r + w + e; - while (sc->sc_open == 0 && sc->sc_active != 0) - msleep(&sc->sc_active, &sc->sc_mtx, 0, "PRIBIO", 0); + if (sc->sc_open == 0) + atomic_clear_int(&sc->sc_active, SC_A_OPEN); + else + atomic_set_int(&sc->sc_active, SC_A_OPEN); + while (sc->sc_open == 0 && (sc->sc_active & SC_A_ACTIVE) != 0) + msleep(&sc->sc_active, &sc->sc_mtx, 0, "g_dev_close", hz / 10); mtx_unlock(&sc->sc_mtx); g_topology_lock(); error = g_access(cp, r, w, e); @@ -707,7 +718,7 @@ g_dev_done(struct bio *bp2) struct g_consumer *cp; struct g_dev_softc *sc; struct bio *bp; - int destroy; + int active; cp = bp2->bio_from; sc = cp->private; @@ -727,17 +738,13 @@ g_dev_done(struct bio *bp2) bp2, bp, bp2->bio_resid, (intmax_t)bp2->bio_completed); } g_destroy_bio(bp2); - destroy = 0; - mtx_lock(&sc->sc_mtx); - if ((--sc->sc_active) == 0) { - if (sc->sc_open == 0) + active = atomic_fetchadd_int(&sc->sc_active, -1) - 1; + if ((active & SC_A_ACTIVE) == 0) { + if ((active & SC_A_OPEN) == 0) wakeup(&sc->sc_active); - if (sc->sc_dev == NULL) - destroy = 1; + if (active & SC_A_DESTROY) + g_post_event(g_dev_destroy, cp, M_NOWAIT, NULL); } - mtx_unlock(&sc->sc_mtx); - if (destroy) - g_post_event(g_dev_destroy, cp, M_NOWAIT, NULL); biodone(bp); } @@ -769,10 +776,8 @@ g_dev_strategy(struct bio *bp) return; } #endif - mtx_lock(&sc->sc_mtx); KASSERT(sc->sc_open > 0, ("Closed device in g_dev_strategy")); - sc->sc_active++; - mtx_unlock(&sc->sc_mtx); + atomic_add_int(&sc->sc_active, 1); for (;;) { /* @@ -810,18 +815,16 @@ g_dev_callback(void *arg) { struct g_consumer *cp; struct g_dev_softc *sc; - int destroy; + int active; cp = arg; sc = cp->private; g_trace(G_T_TOPOLOGY, "g_dev_callback(%p(%s))", cp, cp->geom->name); - mtx_lock(&sc->sc_mtx); sc->sc_dev = NULL; sc->sc_alias = NULL; - destroy = (sc->sc_active == 0); - mtx_unlock(&sc->sc_mtx); - if (destroy) + active = atomic_fetchadd_int(&sc->sc_active, SC_A_DESTROY); + if ((active & SC_A_ACTIVE) == 0) g_post_event(g_dev_destroy, cp, M_WAITOK, NULL); }