From owner-freebsd-arch@FreeBSD.ORG Wed Sep 5 15:10:35 2012 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 61E751065758; Wed, 5 Sep 2012 15:10:35 +0000 (UTC) (envelope-from freebsd@damnhippie.dyndns.org) Received: from duck.symmetricom.us (duck.symmetricom.us [206.168.13.214]) by mx1.freebsd.org (Postfix) with ESMTP id CBC8D8FC08; Wed, 5 Sep 2012 15:10:22 +0000 (UTC) Received: from damnhippie.dyndns.org (daffy.symmetricom.us [206.168.13.218]) by duck.symmetricom.us (8.14.5/8.14.5) with ESMTP id q85FAMIF078927; Wed, 5 Sep 2012 09:10:22 -0600 (MDT) (envelope-from freebsd@damnhippie.dyndns.org) Received: from [172.22.42.240] (revolution.hippie.lan [172.22.42.240]) by damnhippie.dyndns.org (8.14.3/8.14.3) with ESMTP id q85F9xhj042697; Wed, 5 Sep 2012 09:09:59 -0600 (MDT) (envelope-from freebsd@damnhippie.dyndns.org) From: Ian Lepore To: Warner Losh In-Reply-To: <3AFC763F-011C-46B4-B500-FE21B704259F@bsdimp.com> References: <1346689154.1140.601.camel@revolution.hippie.lan> <3AFC763F-011C-46B4-B500-FE21B704259F@bsdimp.com> Content-Type: text/plain; charset="us-ascii" Date: Wed, 05 Sep 2012 09:09:59 -0600 Message-ID: <1346857799.59094.66.camel@revolution.hippie.lan> Mime-Version: 1.0 X-Mailer: Evolution 2.32.1 FreeBSD GNOME Team Port Content-Transfer-Encoding: 7bit Cc: freebsd-arm@freebsd.org, freebsd-mips@freebsd.org, freebsd-arch@freebsd.org Subject: Re: Some busdma stats X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Sep 2012 15:10:35 -0000 On Wed, 2012-09-05 at 08:30 -0600, Warner Losh wrote: > > Regardless of whether we eventually fix every driver to eliminate > > transfers that aren't aligned to cache line boundaries, or somehow > > change the busdma code to automatically bounce unaligned requests, > we > > need efficient allocation of small buffers aligned and sized to > cache > > lines. > > The issue can't be fixed in the busdma code because partial, unaligned > transfers are fine, so long as the calling code avoids the entire > cache line during the transfer. Returning cache-line aligned buffers > from the allocator will do that, of course, but it is also valid for > the code to only use part of the buffer for the transfer. Right. My goal with the dma buffer pool changes isn't some sort of magical automatic fix in the busdma layer, it's just a whittling away of one small roadblock on the path to fixing this stuff. When I first started asking about how we should address these problems, the experts said to keep platform-specific alignment and padding information encapsulated within the busdma layer rather than inventing a new mechanism to export that info to drivers. That implies that drivers should be allocating DMA buffers from busdma instead of allocating big chunks of memory and sub-dividing them into smaller buffers. For that to work, the busdma implementation needs to be able to efficiently allocate buffers that are properly aligned and padded and especially that are guaranteed not to share a cache line with some other unrelated data. The busdma implementation can't get those guarantees from malloc(9), and the alternatives (contigmalloc(), and the kmem_alloc family) only work in page-sized chunks. We're asking drivers to allocate individual buffers of sometimes no more than a few bytes each. So that's all I'm addressing in the patchset I submitted: make sure that when we start fixing drivers to allocate 256 individual 16-byte IO descriptors that it shares with the hardware, that doesn't result in allocating 256 pages of memory. Also, if the request is for BUS_DMA_COHERENT memory, make sure that doesn't result in turning off caching in up to 256 pages that each contain a 16 byte IO buffer and 4080 bytes of unrelated data. -- Ian