From owner-svn-src-all@freebsd.org Fri Jan 26 22:22:30 2018 Return-Path: Delivered-To: svn-src-all@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0E394ECCA66; Fri, 26 Jan 2018 22:22:30 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 3398068A28; Fri, 26 Jan 2018 22:22:28 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id A70E72188E; Fri, 26 Jan 2018 22:22:27 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w0QMMRof038056; Fri, 26 Jan 2018 22:22:27 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w0QMMQXC038055; Fri, 26 Jan 2018 22:22:26 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201801262222.w0QMMQXC038055@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Fri, 26 Jan 2018 22:22:26 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org Subject: svn commit: r328450 - head/sys/dev/ntb X-SVN-Group: head X-SVN-Commit-Author: mav X-SVN-Commit-Paths: head/sys/dev/ntb X-SVN-Commit-Revision: 328450 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-all@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "SVN commit messages for the entire src tree \(except for " user" and " projects" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 Jan 2018 22:22:30 -0000 Author: mav Date: Fri Jan 26 22:22:26 2018 New Revision: 328450 URL: https://svnweb.freebsd.org/changeset/base/328450 Log: Use bus_dmamem_alloc(9) KPI instead of contigmalloc(9). I suppose it should make this code NUMA-aware with recent NUMA drop-in, trying to allocate shared memory buffers from domain closer to NT-bridge. MFC after: 2 weeks Modified: head/sys/dev/ntb/ntb_transport.c Modified: head/sys/dev/ntb/ntb_transport.c ============================================================================== --- head/sys/dev/ntb/ntb_transport.c Fri Jan 26 22:22:21 2018 (r328449) +++ head/sys/dev/ntb/ntb_transport.c Fri Jan 26 22:22:26 2018 (r328450) @@ -182,6 +182,8 @@ struct ntb_transport_mw { size_t xlat_size; size_t buff_size; /* Rx buff is off virt_addr / dma_addr */ + bus_dma_tag_t dma_tag; + bus_dmamap_t dma_map; caddr_t virt_addr; bus_addr_t dma_addr; }; @@ -1147,10 +1149,25 @@ out: NTB_LINK_DOWN_TIMEOUT * hz / 1000, ntb_transport_link_work, nt); } +struct ntb_load_cb_args { + bus_addr_t addr; + int error; +}; + +static void +ntb_load_cb(void *xsc, bus_dma_segment_t *segs, int nsegs, int error) +{ + struct ntb_load_cb_args *cba = (struct ntb_load_cb_args *)xsc; + + if (!(cba->error = error)) + cba->addr = segs[0].ds_addr; +} + static int ntb_set_mw(struct ntb_transport_ctx *nt, int num_mw, size_t size) { struct ntb_transport_mw *mw = &nt->mw_vec[num_mw]; + struct ntb_load_cb_args cba; size_t xlat_size, buff_size; int rc; @@ -1171,30 +1188,36 @@ ntb_set_mw(struct ntb_transport_ctx *nt, int num_mw, s mw->xlat_size = xlat_size; mw->buff_size = buff_size; - mw->virt_addr = contigmalloc(mw->buff_size, M_NTB_T, M_ZERO, 0, - mw->addr_limit, mw->xlat_align, 0); - if (mw->virt_addr == NULL) { + if (bus_dma_tag_create(bus_get_dma_tag(nt->dev), mw->xlat_align, 0, + mw->addr_limit, BUS_SPACE_MAXADDR, + NULL, NULL, mw->buff_size, 1, mw->buff_size, + 0, NULL, NULL, &mw->dma_tag)) { + ntb_printf(0, "Unable to create MW tag of size %zu/%zu\n", + mw->buff_size, mw->xlat_size); + mw->xlat_size = 0; + mw->buff_size = 0; + return (ENOMEM); + } + if (bus_dmamem_alloc(mw->dma_tag, (void **)&mw->virt_addr, + BUS_DMA_WAITOK | BUS_DMA_ZERO, &mw->dma_map)) { + bus_dma_tag_destroy(mw->dma_tag); ntb_printf(0, "Unable to allocate MW buffer of size %zu/%zu\n", mw->buff_size, mw->xlat_size); mw->xlat_size = 0; mw->buff_size = 0; return (ENOMEM); } - /* TODO: replace with bus_space_* functions */ - mw->dma_addr = vtophys(mw->virt_addr); - - /* - * Ensure that the allocation from contigmalloc is aligned as - * requested. XXX: This may not be needed -- brought in for parity - * with the Linux driver. - */ - if (mw->dma_addr % mw->xlat_align != 0) { - ntb_printf(0, - "DMA memory 0x%jx not aligned to BAR size 0x%zx\n", - (uintmax_t)mw->dma_addr, size); - ntb_free_mw(nt, num_mw); + if (bus_dmamap_load(mw->dma_tag, mw->dma_map, mw->virt_addr, + mw->buff_size, ntb_load_cb, &cba, BUS_DMA_NOWAIT) || cba.error) { + bus_dmamem_free(mw->dma_tag, mw->virt_addr, mw->dma_map); + bus_dma_tag_destroy(mw->dma_tag); + ntb_printf(0, "Unable to load MW buffer of size %zu/%zu\n", + mw->buff_size, mw->xlat_size); + mw->xlat_size = 0; + mw->buff_size = 0; return (ENOMEM); } + mw->dma_addr = cba.addr; /* Notify HW the memory location of the receive buffer */ rc = ntb_mw_set_trans(nt->dev, num_mw, mw->dma_addr, mw->xlat_size); @@ -1216,7 +1239,9 @@ ntb_free_mw(struct ntb_transport_ctx *nt, int num_mw) return; ntb_mw_clear_trans(nt->dev, num_mw); - contigfree(mw->virt_addr, mw->xlat_size, M_NTB_T); + bus_dmamap_unload(mw->dma_tag, mw->dma_map); + bus_dmamem_free(mw->dma_tag, mw->virt_addr, mw->dma_map); + bus_dma_tag_destroy(mw->dma_tag); mw->xlat_size = 0; mw->buff_size = 0; mw->virt_addr = NULL;