From owner-freebsd-arch@FreeBSD.ORG Sat Apr 25 20:14:16 2015 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 3FA82CB4 for ; Sat, 25 Apr 2015 20:14:16 +0000 (UTC) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id D4743164A for ; Sat, 25 Apr 2015 20:14:15 +0000 (UTC) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.14.9/8.14.9) with ESMTP id t3PKEADd057498 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Sat, 25 Apr 2015 23:14:10 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.9.2 kib.kiev.ua t3PKEADd057498 Received: (from kostik@localhost) by tom.home (8.14.9/8.14.9/Submit) id t3PKEAXF057497; Sat, 25 Apr 2015 23:14:10 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Sat, 25 Apr 2015 23:14:10 +0300 From: Konstantin Belousov To: Jason Harmening Cc: Svatopluk Kraus , FreeBSD Arch Subject: Re: bus_dmamap_sync() for bounced client buffers from user address space Message-ID: <20150425201410.GP2390@kib.kiev.ua> References: <20150425094152.GE2390@kib.kiev.ua> <553B9E64.8030907@gmail.com> <20150425163444.GL2390@kib.kiev.ua> <553BC9D1.1070502@gmail.com> <20150425172833.GM2390@kib.kiev.ua> <553BD501.4010109@gmail.com> <20150425181846.GN2390@kib.kiev.ua> <553BE12B.4000105@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <553BE12B.4000105@gmail.com> User-Agent: Mutt/1.5.23 (2014-03-12) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.0 X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on tom.home X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 25 Apr 2015 20:14:16 -0000 On Sat, Apr 25, 2015 at 01:47:07PM -0500, Jason Harmening wrote: > > On 04/25/15 13:18, Konstantin Belousov wrote: > > On Sat, Apr 25, 2015 at 12:55:13PM -0500, Jason Harmening wrote: > >> Ah, that looks much better. A few things though: > >> 1) _bus_dmamap_load_ma (note the underscore) is still part of the MI/MD > >> interface, which we tell drivers not to use. It looks like it's > >> implemented for every arch though. Should there be a public and > >> documented bus_dmamap_load_ma ? > > Might be yes. But at least one consumer of the KPI must appear before > > the facility is introduced. > > Could some of the GART/GTT code consume that? Do you mean, by GEM/GTT code ? Indeed, this is interesting and probably workable suggestion. I thought that I would need to provide a special interface from DMAR for the GEM, but your proposal seems to fit. Still, an issue is that the Linux code is structured significantly different, and this code, although isolated, is significant divergent from the upstream. The special DMAR interface is still needed for bhyve, I am slowly working on it. > > >> 2) There is a bus_dmamap_load_ma_triv that's part of the MI interface, > >> but it's not documented, and it seems like it would be suboptimal in > >> certain cases, such as when dmar is enabled. > > When DMAR is enabled, bus_dmamap_load_triv() should not be used. > > It should not be used directly even when not. Drivers should use > > bus_dmamap_load_ma(), and implementation redirects to _triv() if > > needed. > > > > The _triv() is the helper to allow bus_dmamap_load_ma() to exists > > on architectures which cannot implement, on not yet implemented, > > proper page array load op. > Yes, I noticed the same thing. I'm not sure why _triv() is treated as > part of the public API and not prefixed with an underscore and a comment > not to use it in drivers. It is not. We do not claim that a function not starting with '_' is part of the driver KPI. Comment would be nice, indeed. > >> 3) Using bus_dmamap_load_ma would mean always using physcopy for bounce > >> buffers...seems like the sfbufs would slow things down ? > > For amd64, sfbufs are nop, due to the direct map. But, I doubt that > > we can combine bounce buffers and performance in the single sentence. > In fact the amd64 implementation of uiomove_fromphys doesn't use sfbufs > at all thanks to the direct map. sparc64 seems to avoid sfbufs as much > as possible too. I don't know what arm64/aarch64 will be able to use. > Those seem like the platforms where bounce buffering would be the most > likely, along with i386 + PAE. They might still be used on 32-bit > platforms for alignment or devices with < 32-bit address width, but then > those are likely to be old and slow anyway. > > I'm still a bit worried about the slowness of waiting for an sfbuf if > one is needed, but in practice that might not be a big issue. >