From owner-svn-src-head@freebsd.org Mon Aug 15 17:41:35 2016 Return-Path: Delivered-To: svn-src-head@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 82662BBACB9; Mon, 15 Aug 2016 17:41:35 +0000 (UTC) (envelope-from jhb@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 38EE61D04; Mon, 15 Aug 2016 17:41:35 +0000 (UTC) (envelope-from jhb@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id u7FHfYOW059280; Mon, 15 Aug 2016 17:41:34 GMT (envelope-from jhb@FreeBSD.org) Received: (from jhb@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id u7FHfY7a059279; Mon, 15 Aug 2016 17:41:34 GMT (envelope-from jhb@FreeBSD.org) Message-Id: <201608151741.u7FHfY7a059279@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: jhb set sender to jhb@FreeBSD.org using -f From: John Baldwin Date: Mon, 15 Aug 2016 17:41:34 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org Subject: svn commit: r304169 - head/sys/dev/cxgbe/common X-SVN-Group: head MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-head@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: SVN commit messages for the src tree for head/-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Aug 2016 17:41:35 -0000 Author: jhb Date: Mon Aug 15 17:41:34 2016 New Revision: 304169 URL: https://svnweb.freebsd.org/changeset/base/304169 Log: Update mailbox writes to work with VF devices. - Use alternate register locations for the data and control registers for VFs. - Do a dummy read to force the writes to the mailbox data registers to post before the write to the control register on VFs. - Do not check the PCI-e firmware register for errors on VFs. Reviewed by: np Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D7483 Modified: head/sys/dev/cxgbe/common/t4_hw.c Modified: head/sys/dev/cxgbe/common/t4_hw.c ============================================================================== --- head/sys/dev/cxgbe/common/t4_hw.c Mon Aug 15 17:40:05 2016 (r304168) +++ head/sys/dev/cxgbe/common/t4_hw.c Mon Aug 15 17:41:34 2016 (r304169) @@ -289,6 +289,14 @@ int t4_wr_mbox_meat_timeout(struct adapt if ((size & 15) || size > MBOX_LEN) return -EINVAL; + if (adap->flags & IS_VF) { + if (is_t6(adap)) + data_reg = FW_T6VF_MBDATA_BASE_ADDR; + else + data_reg = FW_T4VF_MBDATA_BASE_ADDR; + ctl_reg = VF_CIM_REG(A_CIM_VF_EXT_MAILBOX_CTRL); + } + /* * If we have a negative timeout, that implies that we can't sleep. */ @@ -343,6 +351,22 @@ int t4_wr_mbox_meat_timeout(struct adapt for (i = 0; i < size; i += 8, p++) t4_write_reg64(adap, data_reg + i, be64_to_cpu(*p)); + if (adap->flags & IS_VF) { + /* + * For the VFs, the Mailbox Data "registers" are + * actually backed by T4's "MA" interface rather than + * PL Registers (as is the case for the PFs). Because + * these are in different coherency domains, the write + * to the VF's PL-register-backed Mailbox Control can + * race in front of the writes to the MA-backed VF + * Mailbox Data "registers". So we need to do a + * read-back on at least one byte of the VF Mailbox + * Data registers before doing the write to the VF + * Mailbox Control register. + */ + t4_read_reg(adap, data_reg); + } + CH_DUMP_MBOX(adap, mbox, data_reg); t4_write_reg(adap, ctl_reg, F_MBMSGVALID | V_MBOWNER(X_MBOWNER_FW)); @@ -355,10 +379,13 @@ int t4_wr_mbox_meat_timeout(struct adapt * Loop waiting for the reply; bail out if we time out or the firmware * reports an error. */ - for (i = 0; - !((pcie_fw = t4_read_reg(adap, A_PCIE_FW)) & F_PCIE_FW_ERR) && - i < timeout; - i += ms) { + pcie_fw = 0; + for (i = 0; i < timeout; i += ms) { + if (!(adap->flags & IS_VF)) { + pcie_fw = t4_read_reg(adap, A_PCIE_FW); + if (pcie_fw & F_PCIE_FW_ERR) + break; + } if (sleep_ok) { ms = delay[delay_idx]; /* last element may repeat */ if (delay_idx < ARRAY_SIZE(delay) - 1)