From owner-svn-src-all@freebsd.org Thu May 24 16:31:19 2018 Return-Path: Delivered-To: svn-src-all@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 60D7BEF7698; Thu, 24 May 2018 16:31:19 +0000 (UTC) (envelope-from imp@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 12F2776789; Thu, 24 May 2018 16:31:19 +0000 (UTC) (envelope-from imp@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id E820414AE7; Thu, 24 May 2018 16:31:18 +0000 (UTC) (envelope-from imp@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w4OGVI73066468; Thu, 24 May 2018 16:31:18 GMT (envelope-from imp@FreeBSD.org) Received: (from imp@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w4OGVIN8066467; Thu, 24 May 2018 16:31:18 GMT (envelope-from imp@FreeBSD.org) Message-Id: <201805241631.w4OGVIN8066467@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: imp set sender to imp@FreeBSD.org using -f From: Warner Losh Date: Thu, 24 May 2018 16:31:18 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org Subject: svn commit: r334166 - head/sys/cam/nvme X-SVN-Group: head X-SVN-Commit-Author: imp X-SVN-Commit-Paths: head/sys/cam/nvme X-SVN-Commit-Revision: 334166 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-all@freebsd.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: "SVN commit messages for the entire src tree \(except for " user" and " projects" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 24 May 2018 16:31:19 -0000 Author: imp Date: Thu May 24 16:31:18 2018 New Revision: 334166 URL: https://svnweb.freebsd.org/changeset/base/334166 Log: We can't release the refcount outside of the periph lock. We're dropping the periph lock then dropping the refcount. However, that violates the locking protocol and is racy. This seems to be the cause of weird occasional panics with a bogus assert. Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D15517 Modified: head/sys/cam/nvme/nvme_da.c Modified: head/sys/cam/nvme/nvme_da.c ============================================================================== --- head/sys/cam/nvme/nvme_da.c Thu May 24 16:25:18 2018 (r334165) +++ head/sys/cam/nvme/nvme_da.c Thu May 24 16:31:18 2018 (r334166) @@ -336,6 +336,8 @@ ndaclose(struct disk *dp) while (softc->refcount != 0) cam_periph_sleep(periph, &softc->refcount, PRIBIO, "ndaclose", 1); + KASSERT(softc->outstanding_cmds == 0, + ("nda %d outstanding commands", softc->outstanding_cmds)); cam_periph_unlock(periph); cam_periph_release(periph); return (0); @@ -986,10 +988,11 @@ ndastart(struct cam_periph *periph, union ccb *start_c out: start_ccb->ccb_h.flags |= CAM_UNLOCKED; softc->outstanding_cmds++; - softc->refcount++; + softc->refcount++; /* For submission only */ cam_periph_unlock(periph); xpt_action(start_ccb); cam_periph_lock(periph); + softc->refcount--; /* Submission done */ /* May have more work to do, so ensure we stay scheduled */ ndaschedule(periph); @@ -1085,6 +1088,7 @@ ndadone(struct cam_periph *periph, union ccb *done_ccb bp1 = TAILQ_FIRST(&queue); cam_iosched_bio_complete(softc->cam_iosched, bp1, done_ccb); xpt_release_ccb(done_ccb); + softc->outstanding_cmds--; ndaschedule(periph); cam_periph_unlock(periph); while ((bp2 = TAILQ_FIRST(&queue)) != NULL) { @@ -1100,11 +1104,6 @@ ndadone(struct cam_periph *periph, union ccb *done_ccb biodone(bp2); } } - /* - * Release the periph refcount taken in mdastart() for each CCB. - */ - KASSERT(softc->refcount >= 1, ("ndadone softc %p refcount %d", softc, softc->refcount)); - softc->refcount--; return; } case NDA_CCB_DUMP: