From owner-svn-src-head@freebsd.org Wed Dec 26 10:40:29 2018 Return-Path: Delivered-To: svn-src-head@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 06CA6134C475; Wed, 26 Dec 2018 10:40:29 +0000 (UTC) (envelope-from kadesai@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id ADB6C745EF; Wed, 26 Dec 2018 10:40:28 +0000 (UTC) (envelope-from kadesai@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 9FD43225DA; Wed, 26 Dec 2018 10:40:28 +0000 (UTC) (envelope-from kadesai@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id wBQAeSth035646; Wed, 26 Dec 2018 10:40:28 GMT (envelope-from kadesai@FreeBSD.org) Received: (from kadesai@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id wBQAeRTQ035641; Wed, 26 Dec 2018 10:40:27 GMT (envelope-from kadesai@FreeBSD.org) Message-Id: <201812261040.wBQAeRTQ035641@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: kadesai set sender to kadesai@FreeBSD.org using -f From: Kashyap D Desai Date: Wed, 26 Dec 2018 10:40:27 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org Subject: svn commit: r342530 - head/sys/dev/mpr X-SVN-Group: head X-SVN-Commit-Author: kadesai X-SVN-Commit-Paths: head/sys/dev/mpr X-SVN-Commit-Revision: 342530 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: ADB6C745EF X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-2.98 / 15.00]; local_wl_from(0.00)[FreeBSD.org]; NEURAL_HAM_MEDIUM(-1.00)[-0.999,0]; NEURAL_HAM_SHORT(-0.98)[-0.984,0]; NEURAL_HAM_LONG(-1.00)[-0.999,0]; ASN(0.00)[asn:11403, ipnet:2610:1c1:1::/48, country:US] X-BeenThere: svn-src-head@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: SVN commit messages for the src tree for head/-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Dec 2018 10:40:29 -0000 Author: kadesai Date: Wed Dec 26 10:40:27 2018 New Revision: 342530 URL: https://svnweb.freebsd.org/changeset/base/342530 Log: Added support for NVMe Task Management Following list of changes done in the driver as a part of TM handling on the NVMe drives. Below changes are only applicable on NVMe drives and only when custom NVMe TM handling bit is set to zero by IOC. 1. Issue LUN reset & Target reset TMs with Target reset method field set to Protocol Level reset (0x3), 2. For LUN & target reset TMs use the timeout value as ControllerResetTO value provided by firmware using PCie Device Page 0, 3. If LUN reset fails to terminates the IO then directly escalate to host reset instead of going for target reset TM, 4. For Abort TM use the timeout value as NVMeAbortTO value given by the IOC using Manufacturing Page 11, 5. Log message "PCie Host Reset failed" message up on receiving P Submitted by: Sreekanth Reddy Reviewed by: Kashyap Desai Approved by: ken MFC after: 3 days Sponsored by: Broadcom Inc Modified: head/sys/dev/mpr/mpr_config.c head/sys/dev/mpr/mpr_sas.c head/sys/dev/mpr/mpr_sas.h head/sys/dev/mpr/mpr_sas_lsi.c head/sys/dev/mpr/mprvar.h Modified: head/sys/dev/mpr/mpr_config.c ============================================================================== --- head/sys/dev/mpr/mpr_config.c Wed Dec 26 10:39:34 2018 (r342529) +++ head/sys/dev/mpr/mpr_config.c Wed Dec 26 10:40:27 2018 (r342530) @@ -324,6 +324,137 @@ out: } /** + * mpr_config_get_man_pg11 - obtain manufacturing page 11 + * @sc: per adapter object + * @mpi_reply: reply mf payload returned from firmware + * @config_page: contents of the config page + * Context: sleep. + * + * Returns 0 for success, non-zero for failure. + */ +int +mpr_config_get_man_pg11(struct mpr_softc *sc, Mpi2ConfigReply_t *mpi_reply, + Mpi2ManufacturingPage11_t *config_page) +{ + MPI2_CONFIG_REQUEST *request; + MPI2_CONFIG_REPLY *reply; + struct mpr_command *cm; + MPI2_CONFIG_PAGE_MAN_11 *page = NULL; + int error = 0; + u16 ioc_status; + + mpr_dprint(sc, MPR_TRACE, "%s\n", __func__); + + if ((cm = mpr_alloc_command(sc)) == NULL) { + printf("%s: command alloc failed @ line %d\n", __func__, + __LINE__); + error = EBUSY; + goto out; + } + request = (MPI2_CONFIG_REQUEST *)cm->cm_req; + bzero(request, sizeof(MPI2_CONFIG_REQUEST)); + request->Function = MPI2_FUNCTION_CONFIG; + request->Action = MPI2_CONFIG_ACTION_PAGE_HEADER; + request->Header.PageType = MPI2_CONFIG_PAGETYPE_MANUFACTURING; + request->Header.PageNumber = 11; + request->Header.PageLength = request->Header.PageVersion = 0; + cm->cm_desc.Default.RequestFlags = MPI2_REQ_DESCRIPT_FLAGS_DEFAULT_TYPE; + cm->cm_data = NULL; + error = mpr_wait_command(sc, &cm, 60, CAN_SLEEP); + reply = (MPI2_CONFIG_REPLY *)cm->cm_reply; + if (error || (reply == NULL)) { + /* FIXME */ + /* + * If the request returns an error then we need to do a diag + * reset + */ + printf("%s: request for header completed with error %d", + __func__, error); + error = ENXIO; + goto out; + } + ioc_status = le16toh(reply->IOCStatus) & MPI2_IOCSTATUS_MASK; + bcopy(reply, mpi_reply, sizeof(MPI2_CONFIG_REPLY)); + if (ioc_status != MPI2_IOCSTATUS_SUCCESS) { + /* FIXME */ + /* + * If the request returns an error then we need to do a diag + * reset + */ + printf("%s: header read with error; iocstatus = 0x%x\n", + __func__, ioc_status); + error = ENXIO; + goto out; + } + /* We have to do free and alloc for the reply-free and reply-post + * counters to match - Need to review the reply FIFO handling. + */ + mpr_free_command(sc, cm); + + if ((cm = mpr_alloc_command(sc)) == NULL) { + printf("%s: command alloc failed @ line %d\n", __func__, + __LINE__); + error = EBUSY; + goto out; + } + request = (MPI2_CONFIG_REQUEST *)cm->cm_req; + bzero(request, sizeof(MPI2_CONFIG_REQUEST)); + request->Function = MPI2_FUNCTION_CONFIG; + request->Action = MPI2_CONFIG_ACTION_PAGE_READ_CURRENT; + request->Header.PageType = MPI2_CONFIG_PAGETYPE_MANUFACTURING; + request->Header.PageNumber = 11; + request->Header.PageVersion = mpi_reply->Header.PageVersion; + request->Header.PageLength = mpi_reply->Header.PageLength; + cm->cm_length = le16toh(mpi_reply->Header.PageLength) * 4; + cm->cm_sge = &request->PageBufferSGE; + cm->cm_sglsize = sizeof(MPI2_SGE_IO_UNION); + cm->cm_flags = MPR_CM_FLAGS_SGE_SIMPLE | MPR_CM_FLAGS_DATAIN; + cm->cm_desc.Default.RequestFlags = MPI2_REQ_DESCRIPT_FLAGS_DEFAULT_TYPE; + page = malloc((cm->cm_length), M_MPR, M_ZERO | M_NOWAIT); + if (!page) { + printf("%s: page alloc failed\n", __func__); + error = ENOMEM; + goto out; + } + cm->cm_data = page; + + error = mpr_wait_command(sc, &cm, 60, CAN_SLEEP); + reply = (MPI2_CONFIG_REPLY *)cm->cm_reply; + if (error || (reply == NULL)) { + /* FIXME */ + /* + * If the request returns an error then we need to do a diag + * reset + */ + printf("%s: request for page completed with error %d", + __func__, error); + error = ENXIO; + goto out; + } + ioc_status = le16toh(reply->IOCStatus) & MPI2_IOCSTATUS_MASK; + bcopy(reply, mpi_reply, sizeof(MPI2_CONFIG_REPLY)); + if (ioc_status != MPI2_IOCSTATUS_SUCCESS) { + /* FIXME */ + /* + * If the request returns an error then we need to do a diag + * reset + */ + printf("%s: page read with error; iocstatus = 0x%x\n", + __func__, ioc_status); + error = ENXIO; + goto out; + } + bcopy(page, config_page, MIN(cm->cm_length, + (sizeof(Mpi2ManufacturingPage11_t)))); + +out: + free(page, M_MPR); + if (cm) + mpr_free_command(sc, cm); + return (error); +} + +/** * mpr_base_static_config_pages - static start of day config pages. * @sc: per adapter object * @@ -332,8 +463,9 @@ out: void mpr_base_static_config_pages(struct mpr_softc *sc) { - Mpi2ConfigReply_t mpi_reply; - int retry; + Mpi2ConfigReply_t mpi_reply; + Mpi2ManufacturingPage11_t man_pg11; + int retry, rc; retry = 0; while (mpr_config_get_ioc_pg8(sc, &mpi_reply, &sc->ioc_pg8)) { @@ -352,6 +484,29 @@ mpr_base_static_config_pages(struct mpr_softc *sc) /*FIXME*/ break; } + } + retry = 0; + while ((rc = mpr_config_get_man_pg11(sc, &mpi_reply, &man_pg11))) { + retry++; + if (retry > 5) { + /* We need to Handle this situation */ + /*FIXME*/ + break; + } + } + + if (!rc) { + sc->custom_nvme_tm_handling = (le16toh(man_pg11.AddlFlags2) & + MPI2_MAN_PG11_ADDLFLAGS2_CUSTOM_TM_HANDLING_MASK); + sc->nvme_abort_timeout = man_pg11.NVMeAbortTO; + + /* Minimum NVMe Abort timeout value should be 6 seconds & + * maximum value should be 60 seconds. + */ + if (sc->nvme_abort_timeout < 6) + sc->nvme_abort_timeout = 6; + if (sc->nvme_abort_timeout > 60) + sc->nvme_abort_timeout = 60; } } Modified: head/sys/dev/mpr/mpr_sas.c ============================================================================== --- head/sys/dev/mpr/mpr_sas.c Wed Dec 26 10:39:34 2018 (r342529) +++ head/sys/dev/mpr/mpr_sas.c Wed Dec 26 10:40:27 2018 (r342530) @@ -470,8 +470,14 @@ mprsas_prepare_volume_remove(struct mprsas_softc *sass req->DevHandle = targ->handle; req->TaskType = MPI2_SCSITASKMGMT_TASKTYPE_TARGET_RESET; - /* SAS Hard Link Reset / SATA Link Reset */ - req->MsgFlags = MPI2_SCSITASKMGMT_MSGFLAGS_LINK_RESET; + if (!targ->is_nvme || sc->custom_nvme_tm_handling) { + /* SAS Hard Link Reset / SATA Link Reset */ + req->MsgFlags = MPI2_SCSITASKMGMT_MSGFLAGS_LINK_RESET; + } else { + /* PCIe Protocol Level Reset*/ + req->MsgFlags = + MPI26_SCSITASKMGMT_MSGFLAGS_PROTOCOL_LVL_RST_PCIE; + } cm->cm_targ = targ; cm->cm_data = NULL; @@ -1360,8 +1366,11 @@ mprsas_logical_unit_reset_complete(struct mpr_softc *s "logical unit reset complete for target %u, but still " "have %u command(s), sending target reset\n", targ->tid, cm_count); - mprsas_send_reset(sc, tm, - MPI2_SCSITASKMGMT_TASKTYPE_TARGET_RESET); + if (!targ->is_nvme || sc->custom_nvme_tm_handling) + mprsas_send_reset(sc, tm, + MPI2_SCSITASKMGMT_TASKTYPE_TARGET_RESET); + else + mpr_reinit(sc); } } @@ -1449,7 +1458,7 @@ mprsas_send_reset(struct mpr_softc *sc, struct mpr_com { MPI2_SCSI_TASK_MANAGE_REQUEST *req; struct mprsas_target *target; - int err; + int err, timeout; target = tm->cm_targ; if (target->handle == 0) { @@ -1462,6 +1471,21 @@ mprsas_send_reset(struct mpr_softc *sc, struct mpr_com req->DevHandle = htole16(target->handle); req->TaskType = type; + if (!target->is_nvme || sc->custom_nvme_tm_handling) { + timeout = MPR_RESET_TIMEOUT; + /* + * Target reset method = + * SAS Hard Link Reset / SATA Link Reset + */ + req->MsgFlags = MPI2_SCSITASKMGMT_MSGFLAGS_LINK_RESET; + } else { + timeout = (target->controller_reset_timeout) ? ( + target->controller_reset_timeout) : (MPR_RESET_TIMEOUT); + /* PCIe Protocol Level Reset*/ + req->MsgFlags = + MPI26_SCSITASKMGMT_MSGFLAGS_PROTOCOL_LVL_RST_PCIE; + } + if (type == MPI2_SCSITASKMGMT_TASKTYPE_LOGICAL_UNIT_RESET) { /* XXX Need to handle invalid LUNs */ MPR_SET_LUN(req->LUN, tm->cm_lun); @@ -1472,11 +1496,6 @@ mprsas_send_reset(struct mpr_softc *sc, struct mpr_com tm->cm_complete = mprsas_logical_unit_reset_complete; mprsas_prepare_for_tm(sc, tm, target, tm->cm_lun); } else if (type == MPI2_SCSITASKMGMT_TASKTYPE_TARGET_RESET) { - /* - * Target reset method = - * SAS Hard Link Reset / SATA Link Reset - */ - req->MsgFlags = MPI2_SCSITASKMGMT_MSGFLAGS_LINK_RESET; tm->cm_targ->target_resets++; mpr_dprint(sc, MPR_RECOVERY|MPR_INFO, "Sending target reset to target %u\n", target->tid); @@ -1498,7 +1517,7 @@ mprsas_send_reset(struct mpr_softc *sc, struct mpr_com tm->cm_data = NULL; tm->cm_complete_data = (void *)tm; - callout_reset(&tm->cm_callout, MPR_RESET_TIMEOUT * hz, + callout_reset(&tm->cm_callout, timeout * hz, mprsas_tm_timeout, tm); err = mpr_map_command(sc, tm); @@ -1599,7 +1618,7 @@ mprsas_send_abort(struct mpr_softc *sc, struct mpr_com { MPI2_SCSI_TASK_MANAGE_REQUEST *req; struct mprsas_target *targ; - int err; + int err, timeout; targ = cm->cm_targ; if (targ->handle == 0) { @@ -1627,7 +1646,12 @@ mprsas_send_abort(struct mpr_softc *sc, struct mpr_com tm->cm_targ = cm->cm_targ; tm->cm_lun = cm->cm_lun; - callout_reset(&tm->cm_callout, MPR_ABORT_TIMEOUT * hz, + if (!targ->is_nvme || sc->custom_nvme_tm_handling) + timeout = MPR_ABORT_TIMEOUT; + else + timeout = sc->nvme_abort_timeout; + + callout_reset(&tm->cm_callout, timeout * hz, mprsas_tm_timeout, tm); targ->aborts++; @@ -3328,8 +3352,14 @@ mprsas_action_resetdev(struct mprsas_softc *sassc, uni req->DevHandle = htole16(targ->handle); req->TaskType = MPI2_SCSITASKMGMT_TASKTYPE_TARGET_RESET; - /* SAS Hard Link Reset / SATA Link Reset */ - req->MsgFlags = MPI2_SCSITASKMGMT_MSGFLAGS_LINK_RESET; + if (!targ->is_nvme || sc->custom_nvme_tm_handling) { + /* SAS Hard Link Reset / SATA Link Reset */ + req->MsgFlags = MPI2_SCSITASKMGMT_MSGFLAGS_LINK_RESET; + } else { + /* PCIe Protocol Level Reset*/ + req->MsgFlags = + MPI26_SCSITASKMGMT_MSGFLAGS_PROTOCOL_LVL_RST_PCIE; + } tm->cm_data = NULL; tm->cm_complete = mprsas_resetdev_complete; Modified: head/sys/dev/mpr/mpr_sas.h ============================================================================== --- head/sys/dev/mpr/mpr_sas.h Wed Dec 26 10:39:34 2018 (r342529) +++ head/sys/dev/mpr/mpr_sas.h Wed Dec 26 10:40:27 2018 (r342530) @@ -82,6 +82,7 @@ struct mprsas_target { uint8_t supports_SSU; uint8_t is_nvme; uint32_t MDTS; + uint8_t controller_reset_timeout; }; struct mprsas_softc { Modified: head/sys/dev/mpr/mpr_sas_lsi.c ============================================================================== --- head/sys/dev/mpr/mpr_sas_lsi.c Wed Dec 26 10:39:34 2018 (r342529) +++ head/sys/dev/mpr/mpr_sas_lsi.c Wed Dec 26 10:40:27 2018 (r342530) @@ -683,6 +683,24 @@ skip_fp_send: } break; } + case MPI2_EVENT_PCIE_DEVICE_STATUS_CHANGE: + { + pMpi26EventDataPCIeDeviceStatusChange_t pcie_status_event_data; + pcie_status_event_data = + (pMpi26EventDataPCIeDeviceStatusChange_t)fw_event->event_data; + + switch (pcie_status_event_data->ReasonCode) { + case MPI26_EVENT_PCIDEV_STAT_RC_PCIE_HOT_RESET_FAILED: + { + mpr_printf(sc, "PCIe Host Reset failed on DevHandle " + "0x%x\n", pcie_status_event_data->DevHandle); + break; + } + default: + break; + } + break; + } case MPI2_EVENT_SAS_DEVICE_DISCOVERY_ERROR: { pMpi25EventDataSasDeviceDiscoveryError_t discovery_error_data; @@ -1317,6 +1335,8 @@ mprsas_add_pcie_device(struct mpr_softc *sc, u16 handl targ->connector_name[3] = ((char *)&config_page.ConnectorName)[3]; targ->is_nvme = device_info & MPI26_PCIE_DEVINFO_NVME; targ->MDTS = config_page2.MaximumDataTransferSize; + if (targ->is_nvme) + targ->controller_reset_timeout = config_page2.ControllerResetTO; /* * Assume always TRUE for encl_level_valid because there is no valid * flag for PCIe. Modified: head/sys/dev/mpr/mprvar.h ============================================================================== --- head/sys/dev/mpr/mprvar.h Wed Dec 26 10:39:34 2018 (r342529) +++ head/sys/dev/mpr/mprvar.h Wed Dec 26 10:40:27 2018 (r342530) @@ -97,6 +97,38 @@ typedef uint16_t u16; typedef uint32_t u32; typedef uint64_t u64; +typedef struct _MPI2_CONFIG_PAGE_MAN_11 +{ + MPI2_CONFIG_PAGE_HEADER Header; /* 0x00 */ + U8 FlashTime; /* 0x04 */ + U8 NVTime; /* 0x05 */ + U16 Flag; /* 0x06 */ + U8 RFIoTimeout; /* 0x08 */ + U8 EEDPTagMode; /* 0x09 */ + U8 AWTValue; /* 0x0A */ + U8 Reserve1; /* 0x0B */ + U8 MaxCmdFrames; /* 0x0C */ + U8 Reserve2; /* 0x0D */ + U16 AddlFlags; /* 0x0E */ + U32 SysRefClk; /* 0x10 */ + U64 Reserve3[3]; /* 0x14 */ + U16 AddlFlags2; /* 0x2C */ + U8 AddlFlags3; /* 0x2E */ + U8 Reserve4; /* 0x2F */ + U64 opDebugEnable; /* 0x30 */ + U64 PlDebugEnable; /* 0x38 */ + U64 IrDebugEnable; /* 0x40 */ + U32 BoardPowerRequirement; /* 0x48 */ + U8 NVMeAbortTO; /* 0x4C */ + U8 Reserve5; /* 0x4D */ + U16 Reserve6; /* 0x4E */ + U32 Reserve7[3]; /* 0x50 */ +} MPI2_CONFIG_PAGE_MAN_11, + MPI2_POINTER PTR_MPI2_CONFIG_PAGE_MAN_11, + Mpi2ManufacturingPage11_t, MPI2_POINTER pMpi2ManufacturingPage11_t; + +#define MPI2_MAN_PG11_ADDLFLAGS2_CUSTOM_TM_HANDLING_MASK (0x0010) + /** * struct dev_mapping_table - device mapping information * @physical_id: SAS address for drives or WWID for RAID volumes @@ -471,6 +503,8 @@ struct mpr_softc { char exclude_ids[80]; struct timeval lastfail; + uint8_t custom_nvme_tm_handling; + uint8_t nvme_abort_timeout; }; struct mpr_config_params { @@ -812,6 +846,8 @@ int mpr_config_get_volume_wwid(struct mpr_softc *sc, u int mpr_config_get_raid_pd_pg0(struct mpr_softc *sc, Mpi2ConfigReply_t *mpi_reply, Mpi2RaidPhysDiskPage0_t *config_page, u32 page_address); +int mpr_config_get_man_pg11(struct mpr_softc *sc, Mpi2ConfigReply_t *mpi_reply, + Mpi2ManufacturingPage11_t *config_page); void mprsas_ir_shutdown(struct mpr_softc *sc, int howto); int mpr_reinit(struct mpr_softc *sc);