From owner-svn-src-all@FreeBSD.ORG Mon Mar 17 21:42:32 2014 Return-Path: Delivered-To: svn-src-all@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id C2E3B5CB; Mon, 17 Mar 2014 21:42:32 +0000 (UTC) Received: from svn.freebsd.org (svn.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id A260CD46; Mon, 17 Mar 2014 21:42:32 +0000 (UTC) Received: from svn.freebsd.org ([127.0.1.70]) by svn.freebsd.org (8.14.8/8.14.8) with ESMTP id s2HLgWQ0070041; Mon, 17 Mar 2014 21:42:32 GMT (envelope-from jimharris@svn.freebsd.org) Received: (from jimharris@localhost) by svn.freebsd.org (8.14.8/8.14.8/Submit) id s2HLgV4g070035; Mon, 17 Mar 2014 21:42:31 GMT (envelope-from jimharris@svn.freebsd.org) Message-Id: <201403172142.s2HLgV4g070035@svn.freebsd.org> From: Jim Harris Date: Mon, 17 Mar 2014 21:42:31 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-9@freebsd.org Subject: svn commit: r263273 - stable/9/sys/dev/nvme X-SVN-Group: stable-9 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-all@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: "SVN commit messages for the entire src tree \(except for " user" and " projects" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 17 Mar 2014 21:42:32 -0000 Author: jimharris Date: Mon Mar 17 21:42:31 2014 New Revision: 263273 URL: http://svnweb.freebsd.org/changeset/base/263273 Log: MFC r254302: Send a shutdown notification in the driver unload path, to ensure notification gets sent in cases where system shuts down with driver unloaded. Sponsored by: Intel Modified: stable/9/sys/dev/nvme/nvme.c stable/9/sys/dev/nvme/nvme.h stable/9/sys/dev/nvme/nvme_ctrlr.c stable/9/sys/dev/nvme/nvme_private.h Directory Properties: stable/9/sys/ (props changed) stable/9/sys/dev/ (props changed) Modified: stable/9/sys/dev/nvme/nvme.c ============================================================================== --- stable/9/sys/dev/nvme/nvme.c Mon Mar 17 20:42:45 2014 (r263272) +++ stable/9/sys/dev/nvme/nvme.c Mon Mar 17 21:42:31 2014 (r263273) @@ -157,30 +157,14 @@ nvme_shutdown(void) { device_t *devlist; struct nvme_controller *ctrlr; - union cc_register cc; - union csts_register csts; int dev, devcount; if (devclass_get_devices(nvme_devclass, &devlist, &devcount)) return; for (dev = 0; dev < devcount; dev++) { - /* - * Only notify controller of shutdown when a real shutdown is - * in process, not when a module unload occurs. It seems at - * least some controllers (Chatham at least) don't let you - * re-enable the controller after shutdown notification has - * been received. - */ ctrlr = DEVICE2SOFTC(devlist[dev]); - cc.raw = nvme_mmio_read_4(ctrlr, cc); - cc.bits.shn = NVME_SHN_NORMAL; - nvme_mmio_write_4(ctrlr, cc, cc.raw); - csts.raw = nvme_mmio_read_4(ctrlr, csts); - while (csts.bits.shst != NVME_SHST_COMPLETE) { - DELAY(5); - csts.raw = nvme_mmio_read_4(ctrlr, csts); - } + nvme_ctrlr_shutdown(ctrlr); } free(devlist, M_TEMP); Modified: stable/9/sys/dev/nvme/nvme.h ============================================================================== --- stable/9/sys/dev/nvme/nvme.h Mon Mar 17 20:42:45 2014 (r263272) +++ stable/9/sys/dev/nvme/nvme.h Mon Mar 17 21:42:31 2014 (r263273) @@ -170,27 +170,30 @@ struct nvme_registers union cap_lo_register cap_lo; union cap_hi_register cap_hi; - uint32_t vs; /* version */ - uint32_t intms; /* interrupt mask set */ - uint32_t intmc; /* interrupt mask clear */ + uint32_t vs; /* version */ + uint32_t intms; /* interrupt mask set */ + uint32_t intmc; /* interrupt mask clear */ /** controller configuration */ union cc_register cc; - uint32_t reserved1; - uint32_t csts; /* controller status */ - uint32_t reserved2; + uint32_t reserved1; + + /** controller status */ + union csts_register csts; + + uint32_t reserved2; /** admin queue attributes */ union aqa_register aqa; - uint64_t asq; /* admin submission queue base addr */ - uint64_t acq; /* admin completion queue base addr */ - uint32_t reserved3[0x3f2]; + uint64_t asq; /* admin submission queue base addr */ + uint64_t acq; /* admin completion queue base addr */ + uint32_t reserved3[0x3f2]; struct { - uint32_t sq_tdbl; /* submission queue tail doorbell */ - uint32_t cq_hdbl; /* completion queue head doorbell */ + uint32_t sq_tdbl; /* submission queue tail doorbell */ + uint32_t cq_hdbl; /* completion queue head doorbell */ } doorbell[1] __packed; } __packed; Modified: stable/9/sys/dev/nvme/nvme_ctrlr.c ============================================================================== --- stable/9/sys/dev/nvme/nvme_ctrlr.c Mon Mar 17 20:42:45 2014 (r263272) +++ stable/9/sys/dev/nvme/nvme_ctrlr.c Mon Mar 17 21:42:31 2014 (r263273) @@ -1177,6 +1177,21 @@ nvme_ctrlr_destruct(struct nvme_controll { int i; + /* + * Notify the controller of a shutdown, even though this is due to + * a driver unload, not a system shutdown (this path is not invoked + * during shutdown). This ensures the controller receives a + * shutdown notification in case the system is shutdown before + * reloading the driver. + * + * Chatham does not let you re-enable the controller after shutdown + * notification has been received, so do not send it in this case. + * This is OK because Chatham does not depend on the shutdown + * notification anyways. + */ + if (pci_get_devid(ctrlr->dev) != CHATHAM_PCI_ID) + nvme_ctrlr_shutdown(ctrlr); + nvme_ctrlr_disable(ctrlr); taskqueue_free(ctrlr->taskqueue); @@ -1223,6 +1238,26 @@ nvme_ctrlr_destruct(struct nvme_controll } void +nvme_ctrlr_shutdown(struct nvme_controller *ctrlr) +{ + union cc_register cc; + union csts_register csts; + int ticks = 0; + + cc.raw = nvme_mmio_read_4(ctrlr, cc); + cc.bits.shn = NVME_SHN_NORMAL; + nvme_mmio_write_4(ctrlr, cc, cc.raw); + csts.raw = nvme_mmio_read_4(ctrlr, csts); + while ((csts.bits.shst != NVME_SHST_COMPLETE) && (ticks++ < 5*hz)) { + pause("nvme shn", 1); + csts.raw = nvme_mmio_read_4(ctrlr, csts); + } + if (csts.bits.shst != NVME_SHST_COMPLETE) + nvme_printf(ctrlr, "did not complete shutdown within 5 seconds " + "of notification\n"); +} + +void nvme_ctrlr_submit_admin_request(struct nvme_controller *ctrlr, struct nvme_request *req) { Modified: stable/9/sys/dev/nvme/nvme_private.h ============================================================================== --- stable/9/sys/dev/nvme/nvme_private.h Mon Mar 17 20:42:45 2014 (r263272) +++ stable/9/sys/dev/nvme/nvme_private.h Mon Mar 17 21:42:31 2014 (r263273) @@ -437,6 +437,7 @@ void nvme_completion_poll_cb(void *arg, int nvme_ctrlr_construct(struct nvme_controller *ctrlr, device_t dev); void nvme_ctrlr_destruct(struct nvme_controller *ctrlr, device_t dev); +void nvme_ctrlr_shutdown(struct nvme_controller *ctrlr); int nvme_ctrlr_hw_reset(struct nvme_controller *ctrlr); void nvme_ctrlr_reset(struct nvme_controller *ctrlr); /* ctrlr defined as void * to allow use with config_intrhook. */