From owner-freebsd-usb@FreeBSD.ORG Tue Jan 13 20:25:05 2015 Return-Path: Delivered-To: freebsd-usb@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id EE84378F for ; Tue, 13 Jan 2015 20:25:04 +0000 (UTC) Received: from mho-01-ewr.mailhop.org (mho-03-ewr.mailhop.org [204.13.248.66]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id AA4B181C for ; Tue, 13 Jan 2015 20:25:04 +0000 (UTC) Received: from [73.34.117.227] (helo=ilsoft.org) by mho-01-ewr.mailhop.org with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.72) (envelope-from ) id 1YB81O-000ALu-6l; Tue, 13 Jan 2015 20:25:02 +0000 Received: from revolution.hippie.lan (revolution.hippie.lan [172.22.42.240]) by ilsoft.org (8.14.9/8.14.9) with ESMTP id t0DKP0c1066514; Tue, 13 Jan 2015 13:25:00 -0700 (MST) (envelope-from ian@freebsd.org) X-Mail-Handler: Dyn Standard SMTP by Dyn X-Originating-IP: 73.34.117.227 X-Report-Abuse-To: abuse@dyndns.com (see http://www.dyndns.com/services/sendlabs/outbound_abuse.html for abuse reporting information) X-MHO-User: U2FsdGVkX1/j88sO821jTKSGZB1ysbRR Message-ID: <1421180700.14601.209.camel@freebsd.org> Subject: Re: usb_pc_cpu_flush From: Ian Lepore To: Hans Petter Selasky Date: Tue, 13 Jan 2015 13:25:00 -0700 In-Reply-To: <54B54D4D.3010805@selasky.org> References: <1419359192795-5975583.post@n5.nabble.com> <5499E734.1070507@selasky.org> <1419392511197-5975691.post@n5.nabble.com> <549A811D.3060204@selasky.org> <1419416870924-5975752.post@n5.nabble.com> <1419423740820-5975763.post@n5.nabble.com> <549AB711.8070005@selasky.org> <1419431704871-5975773.post@n5.nabble.com> <549BF430.8000207@selasky.org> <1419877515606-5976832.post@n5.nabble.com> <1421133295061-5980199.post@n5.nabble.com> <1421160576.14601.175.camel@freebsd.org> <54B53956.4090708@selasky.org> <1421163656.14601.184.camel@freebsd.org> <54B54073.6000409@selasky.org> <1421166591.14601.195.camel@freebsd.org> <54B54D4D.3010805@selasky.org> Content-Type: text/plain; charset="us-ascii" X-Mailer: Evolution 3.12.8 FreeBSD GNOME Team Port Mime-Version: 1.0 Content-Transfer-Encoding: 7bit Cc: kott , freebsd-usb@freebsd.org X-BeenThere: freebsd-usb@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: FreeBSD support for USB List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Jan 2015 20:25:05 -0000 On Tue, 2015-01-13 at 17:52 +0100, Hans Petter Selasky wrote: > On 01/13/15 17:29, Ian Lepore wrote: > > On Tue, 2015-01-13 at 16:57 +0100, Hans Petter Selasky wrote: > >> On 01/13/15 16:40, Ian Lepore wrote: > >>> On Tue, 2015-01-13 at 16:27 +0100, Hans Petter Selasky wrote: > >>>> On 01/13/15 15:49, Ian Lepore wrote: > >>>>> On Tue, 2015-01-13 at 00:14 -0700, kott wrote: > >>>>>> Yes with cache disabled, this problem is not seen. Seems to be with a issue > >>>>>> with l2 cache. > >>>>>> Thanks kott > >>>>> > >>>>> Except that there are no known problems with l2 cache on armv7 right > >>>>> now. There are known problems with the USB driver using the busdma > >>>>> routines incorrectly, which accidentally works okay on x86 platforms but > >>>>> likely not so well on others. > >>>>> > >>>> > >>>> Hi, > >>>> > >>>> If there is a problem it is in "usb_pc_cpu_flush()" or > >>>> "usb_pc_cpu_invalidate()": > >>>> > >>>> void > >>>> usb_pc_cpu_flush(struct usb_page_cache *pc) > >>>> { > >>>> if (pc->page_offset_end == pc->page_offset_buf) { > >>>> /* nothing has been loaded into this page cache! */ > >>>> return; > >>>> } > >>>> bus_dmamap_sync(pc->tag, pc->map, BUS_DMASYNC_PREWRITE); > >>>> } > >>>> > >>>> USB has a very simple DMA sync language, either flush or invalidate. > >>>> These are used correctly from what I can see with regard to the FreeBSD > >>>> USB specification. > >>>> (unless a simple restart somehow fixes the problem) > >>>> If the "usb_pc_cpu_flush()" function does not cause the CPU cache to be > >>>> written to RAM before the function returns, please let me know. > >>>> > >>>> --HPS > >>> > >>> You have an incomplete concept of how busdma sync operations work. It > >>> isn't a simple "cpu cache written to ram" operation, there are bounce > >>> buffers and other complexities involved that require that the sync > >>> operations be done at the correct time in the correct order, and the > >>> current usb driver doesn't do that. Instead it does things like > >>> > >>> bus_dmamap_sync(pc->tag, pc->map, BUS_DMASYNC_POSTREAD); > >>> bus_dmamap_sync(pc->tag, pc->map, BUS_DMASYNC_PREREAD); > >>> > >>> And that's just nonsense that will lead to problems like delivering > >>> random buffer garbage to/from a device. > >>> > >>> To the degree that USB works at all on non-x86 platforms it works by > >>> accident. Usually. Except when it doesn't. > >>> > >>> -- Ian > >>> > >> > >> Hi, > >> > >> Bounce buffers are perfectly fine with USB as long as the busdma does > >> what it is told. If there is no easy way to do a simple "cache flush" or > >> "cache invalide" or bounce buffer "flush" or bounce buffer "invalidate" > >> multiple times in a row, then busdma cannot co-exist with USB. It is not > >> because I'm stubborn, but because of the way USB DMA controllers are > >> designed. > >> > >> With USB chipsets we sometimes need to read the RAM area for a single > >> buffer multiple times to poll for updates. From what I've been told in > >> the past BUSDMA does. > >> > >> --HPS > >> > >> --HPS > >> > >> --HPS > >> > > > > And so we reach the same old impasse, where you declare that USB is > > special and doesn't have to behave like other drivers, even though it is > > in no way unique in terms of having things like concurrent shared access > > to descriptor memory, something that virtually every modern NIC has. > > > > Hi, > > Can you give an example of a NIC driver which you consider a good > example which use DMA both for data (not only mbufs) and the control > path and use busdma as you consider to be correct? > > --HPS dev/ffec/if_ffec.c. I'm not happy with the fact that it takes two calls (a PRE and a POST) to accomplish a single action, but that's the right way to do things in the current busdma world, PRE and POST operations need to be paired. I think we need new busdma support for shared concurrent access descriptors, because it's a type of dma that just isn't supported well by the existing API. There should be a new flag for bus_dmamem_alloc() that indicates the memory is going to be used for such shared access (because some platforms may be able to provide memory that's mapped optimally for such situations), and there should be new sync ops that don't require a pair of calls to accomplish a single action. All of this is in the context of shared descriptor memory. Regular IO buffers should just use the proper sequence of PRE and POST syncs (and most especially should *never* do POSTREAD before PREREAD like the current usb_pc_cpu_invalidate() does, because with bounce buffers that will just not work right). -- Ian