From owner-svn-src-all@FreeBSD.ORG Tue May 20 22:43:18 2014 Return-Path: Delivered-To: svn-src-all@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 964F4224; Tue, 20 May 2014 22:43:18 +0000 (UTC) Received: from svn.freebsd.org (svn.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 773EC2BD7; Tue, 20 May 2014 22:43:18 +0000 (UTC) Received: from svn.freebsd.org ([127.0.1.70]) by svn.freebsd.org (8.14.8/8.14.8) with ESMTP id s4KMhIUh033127; Tue, 20 May 2014 22:43:18 GMT (envelope-from scottl@svn.freebsd.org) Received: (from scottl@localhost) by svn.freebsd.org (8.14.8/8.14.8/Submit) id s4KMhIRu033126; Tue, 20 May 2014 22:43:18 GMT (envelope-from scottl@svn.freebsd.org) Message-Id: <201405202243.s4KMhIRu033126@svn.freebsd.org> From: Scott Long Date: Tue, 20 May 2014 22:43:18 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org Subject: svn commit: r266481 - head/sys/x86/x86 X-SVN-Group: head MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-all@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: "SVN commit messages for the entire src tree \(except for " user" and " projects" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 May 2014 22:43:18 -0000 Author: scottl Date: Tue May 20 22:43:17 2014 New Revision: 266481 URL: http://svnweb.freebsd.org/changeset/base/266481 Log: Old PCIe implementations cannot allow a DMA transfer to cross a 4GB boundary. This was addressed several years ago by creating a parent tag hierarchy for the root buses that set the boundary restriction for appropriate buses and allowed child deviced to inherit it. Somewhere along the way, this restriction was turned into a case for marking the tag as a candidate for needing bounce buffers, instead of just splitting the segment along the boundary line. This flag also causes all maps associated with this tag to be non-NULL, which in turn causes bus_dmamap_sync() to take the slow path of function pointer indirection to discover that there's no bouncing work to do. The end result is a lot of pages set aside in bounce pools that will never be used, and a slow path for data buffers in nearly every DMA-capable PCIe device. For example, our workload at Netflix was spending nearly 1% of all CPU time going through this slow path. Fix this problem by being more selective about when to set the COULD_BOUNCE flag. Only set it when the boundary restriction exists and the consumer cannot do more than a single DMA segment at once. This fixes the case of dynamic buffers (mbufs, bio's) but doesn't address static buffers allocated from bus_dmamem_alloc(). That case will be addressed in the future. For those interested, this was discovered thanks to Dtrace Flame Graphs. Discussed with: jhb, kib Obtained from: Netflix, Inc. MFC after: 3 days Modified: head/sys/x86/x86/busdma_bounce.c Modified: head/sys/x86/x86/busdma_bounce.c ============================================================================== --- head/sys/x86/x86/busdma_bounce.c Tue May 20 22:11:52 2014 (r266480) +++ head/sys/x86/x86/busdma_bounce.c Tue May 20 22:43:17 2014 (r266481) @@ -172,12 +172,35 @@ bounce_bus_dma_tag_create(bus_dma_tag_t newtag->map_count = 0; newtag->segments = NULL; + /* + * Bouncing might be needed if there's a filter. + * XXX Filters are likely broken as there's no way to + * guarantee that bounce pages will also satisfy the + * filter requirement. + */ if (parent != NULL && ((newtag->common.filter != NULL) || ((parent->common.flags & BUS_DMA_COULD_BOUNCE) != 0))) newtag->common.flags |= BUS_DMA_COULD_BOUNCE; - if (newtag->common.lowaddr < ptoa((vm_paddr_t)Maxmem) || - newtag->common.alignment > 1) + /* + * Bouncing might be needed if there's an upper memory + * restriction. + */ + if (newtag->common.lowaddr < ptoa((vm_paddr_t)Maxmem)) + newtag->common.flags |= BUS_DMA_COULD_BOUNCE; + + /* + * Bouncing might be needed if there's an alignment + * restriction that can't be satisfied by breaking up + * the segment. + * XXX Need to consider non-natural alignment. + * XXX Static allocations that tie to bus_dmamem_alloc() + * will likely pass this test and be penalized with + * the COULD_BOUNCE flag. Should probably have + * bus_dmamem_alloc() clear this flag. + */ + if ((newtag->common.nsegments <= 1) && + (newtag->common.alignment > 1)) newtag->common.flags |= BUS_DMA_COULD_BOUNCE; if (((newtag->common.flags & BUS_DMA_COULD_BOUNCE) != 0) &&