From owner-svn-src-all@FreeBSD.ORG Mon Nov 23 20:44:34 2009 Return-Path: Delivered-To: svn-src-all@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A494E106566C; Mon, 23 Nov 2009 20:44:34 +0000 (UTC) (envelope-from pyunyh@gmail.com) Received: from mail-yx0-f171.google.com (mail-yx0-f171.google.com [209.85.210.171]) by mx1.freebsd.org (Postfix) with ESMTP id D06858FC0A; Mon, 23 Nov 2009 20:44:33 +0000 (UTC) Received: by yxe1 with SMTP id 1so5203304yxe.3 for ; Mon, 23 Nov 2009 12:44:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:received:from:date:to:cc :subject:message-id:reply-to:references:mime-version:content-type :content-disposition:in-reply-to:user-agent; bh=iWauVuorTNgerR6tRj9goq2sl7Hg+RnaEYdUoyCd0f0=; b=dDVBuWMGpr4u80kOqV/9jfy3EL9i2AetZsBXBacMUL8vv0fJaziXAAYPGUwcKqYF8+ Dqi4Lh2uhB2GJzl/Jm9WyUijEFQfi4rRksIZ2KLSA+bEVy3SDyvIWKH53rPnvf0g3/QQ L3WLPGq6E1VSGRn+98mP61MXv3ztQefm29yVk= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=from:date:to:cc:subject:message-id:reply-to:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; b=ptZkqk0e1AFyoUvsES21mh+fQMrTkGeb57tzsJaRCkpDvF/jk2X+opobB7rd3D0GsF DfPEnHC1HdLjv7gzcpYr7wkU5KATa3fBqBkwgLfXY+mG4rPOgn4DqX+q8Aqd2cFOlbMj 8wRgF3+0Qhu/en5CJe7C+vZ3OqnViGa1mo7CQ= Received: by 10.101.166.2 with SMTP id t2mr4294380ano.9.1259009069276; Mon, 23 Nov 2009 12:44:29 -0800 (PST) Received: from pyunyh@gmail.com ([174.35.1.224]) by mx.google.com with ESMTPS id 9sm1368971ywe.56.2009.11.23.12.44.26 (version=TLSv1/SSLv3 cipher=RC4-MD5); Mon, 23 Nov 2009 12:44:28 -0800 (PST) Received: by pyunyh@gmail.com (sSMTP sendmail emulation); Mon, 23 Nov 2009 12:44:00 -0800 From: Pyun YongHyeon Date: Mon, 23 Nov 2009 12:44:00 -0800 To: Scott Long Message-ID: <20091123204400.GB1214@michelle.cdnetworks.com> References: <200911222050.nAMKoRYh029141@svn.freebsd.org> <0298EC7E-26D5-460A-9EBE-5ABEAE8B21BD@samsco.org> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="AhhlLboLdkugWU4S" Content-Disposition: inline In-Reply-To: <0298EC7E-26D5-460A-9EBE-5ABEAE8B21BD@samsco.org> User-Agent: Mutt/1.4.2.3i Cc: svn-src-head@freebsd.org, svn-src-all@freebsd.org, src-committers@freebsd.org, Pyun YongHyeon Subject: Re: svn commit: r199670 - head/sys/dev/bge X-BeenThere: svn-src-all@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: pyunyh@gmail.com List-Id: "SVN commit messages for the entire src tree \(except for " user" and " projects" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Nov 2009 20:44:34 -0000 --AhhlLboLdkugWU4S Content-Type: text/plain; charset=us-ascii Content-Disposition: inline On Sun, Nov 22, 2009 at 04:48:18PM -0700, Scott Long wrote: > By definition, PCIe can't transfer across 4GB boundaries. It's not a > bug specific to Broadcom. If you're loading dynamic buffers (i.e. > mbufs), setting an appropriate boundary value in the tag will allow > busdma to take care of this. If you're allocating static buffers, > busdma won't honor this. But what you've done here is best anyways; Last time I tried boundary value for static buffer in dma tag it didn't honor the boundary value so I blindly thought boundary value also is not honored for dynamic buffers. It would be great if this can be documented in man page. > control buffers that are going to be frequently transferred are best > kept in the lower 4GB of the address space. It simplifies PCIe > handling, and it's significantly faster on PCI/PCI-X. So I'd suggest > making this the rule rather than the exception in the driver. > If this is the case it would also affect other drivers(em(4) and igb(4) etc). As John said it would be even better this could be automatically handled in bus_dma. Patching every driver in tree to reflect this would be pain, I guess. Anyway I've made a new patch. Would you review this? --AhhlLboLdkugWU4S Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename="bge.dma.diff2" --- if_bge.c.orig 2009-11-22 13:45:55.000000000 -0800 +++ if_bge.c 2009-11-23 12:20:07.000000000 -0800 @@ -2028,7 +2028,8 @@ bus_dma_tag_destroy(sc->bge_cdata.bge_rx_mtag); if (sc->bge_cdata.bge_tx_mtag) bus_dma_tag_destroy(sc->bge_cdata.bge_tx_mtag); - + if (sc->bge_cdata.bge_mbuf_tag) + bus_dma_tag_destroy(sc->bge_cdata.bge_mbuf_tag); /* Destroy standard RX ring. */ if (sc->bge_cdata.bge_rx_std_ring_map) @@ -2125,16 +2126,24 @@ sc = device_get_softc(dev); - lowaddr = BUS_SPACE_MAXADDR; - if ((sc->bge_flags & BGE_FLAG_40BIT_BUG) != 0) - lowaddr = BGE_DMA_MAXADDR; - if ((sc->bge_flags & BGE_FLAG_4G_BNDRY_BUG) != 0) - lowaddr = BUS_SPACE_MAXADDR_32BIT; /* * Allocate the parent bus DMA tag appropriate for PCI. + * All controllers that are not 5755 or higher have 4GB + * boundary DMA bug. + * Whenever an address crosses a multiple of the 4GB boundary + * (including 4GB, 8Gb, 12Gb, etc.) and makes the transition + * from 0xX_FFFF_FFFF to 0x(X+1)_0000_0000 an internal DMA + * state machine will lockup and cause the device to hang. + * According to Scott Long, PCIe can't transfer across 4GB + * boundaries by definition. And control buffers that are + * going to frequently transferred are best kept in the lower + * 4GB of the address space. It simplifies PCIe handling and + * it's significantly faster on PCI/PCI-X. So limit the + * DMA address to be within 4GB address for all control + * buffers(TX/RX/RX return/status block/statistics block etc). */ error = bus_dma_tag_create(bus_get_dma_tag(sc->bge_dev), - 1, 0, lowaddr, BUS_SPACE_MAXADDR, NULL, + 1, 0, BUS_SPACE_MAXADDR_32BIT, BUS_SPACE_MAXADDR, NULL, NULL, BUS_SPACE_MAXSIZE_32BIT, 0, BUS_SPACE_MAXSIZE_32BIT, 0, NULL, NULL, &sc->bge_cdata.bge_parent_tag); @@ -2143,6 +2152,23 @@ "could not allocate parent dma tag\n"); return (ENOMEM); } + /* + * Set 4GB boundary for mbufs such that it workarounds DMA bug + * of pre BCM5755 controllers. + */ + lowaddr = BUS_SPACE_MAXADDR; + if ((sc->bge_flags & BGE_FLAG_40BIT_BUG) != 0) + lowaddr = BGE_DMA_MAXADDR; + error = bus_dma_tag_create(bus_get_dma_tag(sc->bge_dev), + 1, BUS_SPACE_MAXSIZE_32BIT, lowaddr, BUS_SPACE_MAXADDR, + NULL, NULL, BUS_SPACE_MAXSIZE_32BIT, 0, BUS_SPACE_MAXSIZE_32BIT, + 0, NULL, NULL, &sc->bge_cdata.bge_mbuf_tag); + + if (error != 0) { + device_printf(sc->bge_dev, + "could not allocate mbuf dma tag\n"); + return (ENOMEM); + } /* * Create tag for Tx mbufs. @@ -2154,7 +2180,7 @@ txsegsz = MCLBYTES; txmaxsegsz = MCLBYTES * BGE_NSEG_NEW; } - error = bus_dma_tag_create(sc->bge_cdata.bge_parent_tag, 1, + error = bus_dma_tag_create(sc->bge_cdata.bge_mbuf_tag, 1, 0, BUS_SPACE_MAXADDR, BUS_SPACE_MAXADDR, NULL, NULL, txmaxsegsz, BGE_NSEG_NEW, txsegsz, 0, NULL, NULL, &sc->bge_cdata.bge_tx_mtag); @@ -2167,7 +2193,7 @@ /* * Create tag for Rx mbufs. */ - error = bus_dma_tag_create(sc->bge_cdata.bge_parent_tag, 1, 0, + error = bus_dma_tag_create(sc->bge_cdata.bge_mbuf_tag, 1, 0, BUS_SPACE_MAXADDR, BUS_SPACE_MAXADDR, NULL, NULL, MCLBYTES, 1, MCLBYTES, 0, NULL, NULL, &sc->bge_cdata.bge_rx_mtag); @@ -2607,17 +2633,6 @@ } /* - * All controllers that are not 5755 or higher have 4GB - * boundary DMA bug. - * Whenever an address crosses a multiple of the 4GB boundary - * (including 4GB, 8Gb, 12Gb, etc.) and makes the transition - * from 0xX_FFFF_FFFF to 0x(X+1)_0000_0000 an internal DMA - * state machine will lockup and cause the device to hang. - */ - if (BGE_IS_5755_PLUS(sc) == 0) - sc->bge_flags |= BGE_FLAG_4G_BNDRY_BUG; - - /* * We could possibly check for BCOM_DEVICEID_BCM5788 in bge_probe() * but I do not know the DEVICEID for the 5788M. */ --- if_bgereg.h.orig 2009-11-22 13:16:30.000000000 -0800 +++ if_bgereg.h 2009-11-23 11:53:56.000000000 -0800 @@ -2537,6 +2537,7 @@ */ struct bge_chain_data { bus_dma_tag_t bge_parent_tag; + bus_dma_tag_t bge_mbuf_tag; bus_dma_tag_t bge_rx_std_ring_tag; bus_dma_tag_t bge_rx_jumbo_ring_tag; bus_dma_tag_t bge_rx_return_ring_tag; @@ -2612,7 +2613,6 @@ #define BGE_FLAG_575X_PLUS 0x00008000 #define BGE_FLAG_5755_PLUS 0x00010000 #define BGE_FLAG_40BIT_BUG 0x00020000 -#define BGE_FLAG_4G_BNDRY_BUG 0x00040000 #define BGE_FLAG_RX_ALIGNBUG 0x00100000 #define BGE_FLAG_NO_3LED 0x00200000 #define BGE_FLAG_ADC_BUG 0x00400000 --AhhlLboLdkugWU4S--