From owner-freebsd-arch@FreeBSD.ORG Sun Aug 26 23:13:35 2012 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id BC340106566C for ; Sun, 26 Aug 2012 23:13:35 +0000 (UTC) (envelope-from imp@bsdimp.com) Received: from mail-iy0-f182.google.com (mail-iy0-f182.google.com [209.85.210.182]) by mx1.freebsd.org (Postfix) with ESMTP id 6AFE18FC19 for ; Sun, 26 Aug 2012 23:13:34 +0000 (UTC) Received: by ialo14 with SMTP id o14so8778790ial.13 for ; Sun, 26 Aug 2012 16:13:34 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=sender:subject:mime-version:content-type:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to:x-mailer :x-gm-message-state; bh=ljCRbjZ9ASHSXhbZZlvbTJ/BcuQ0M4nJMzidj/rAYzc=; b=gQLUWH8H4WF5+fAkTGCGwmQh27HEh6WLHshRdWbQED0en6El9jNVvpdvA809HP0ZqI fBXuif2+qu2KD8ffo5dA3tIopHA7VcTA1MNuBSAZsgN+kNnEEnAcRSvwQM6u1SZAJ1k7 US+tDP8brfdR/8yUlJfgHlwvWLvt5ajhh2o49bh+vcOY+hZXJQHnr9aqKomDiKQguDRe 1HEMxnjEteSGX+YP7Dnf5Djpjfr7bIIsIl0M92sxNaINxm4FbzqeKTq4uL86m4A4zWOa pO64dazQqKMgoO75P0sOKQZsTwHJB4SCazOeP/RG3dPzF8+h5l8iDGyNltQgNlWWZOur E9ug== Received: by 10.50.187.229 with SMTP id fv5mr2371955igc.57.1346022814665; Sun, 26 Aug 2012 16:13:34 -0700 (PDT) Received: from 63.imp.bsdimp.com (50-78-194-198-static.hfc.comcastbusiness.net. [50.78.194.198]) by mx.google.com with ESMTPS id df1sm16929663igc.10.2012.08.26.16.13.32 (version=TLSv1/SSLv3 cipher=OTHER); Sun, 26 Aug 2012 16:13:33 -0700 (PDT) Sender: Warner Losh Mime-Version: 1.0 (Apple Message framework v1084) Content-Type: text/plain; charset=us-ascii From: Warner Losh In-Reply-To: <1346005507.1140.69.camel@revolution.hippie.lan> Date: Sun, 26 Aug 2012 17:13:31 -0600 Content-Transfer-Encoding: quoted-printable Message-Id: <10307B47-13F3-45C0-87F7-66FD3ACA3F86@bsdimp.com> References: <1345757300.27688.535.camel@revolution.hippie.lan> <3A08EB08-2BBF-4B0F-97F2-A3264754C4B7@bsdimp.com> <1345763393.27688.578.camel@revolution.hippie.lan> <1345765503.27688.602.camel@revolution.hippie.lan> <1345766109.27688.606.camel@revolution.hippie.lan> <1346002922.1140.56.camel@revolution.hippie.lan> <1346005507.1140.69.camel@revolution.hippie.lan> To: Ian Lepore X-Mailer: Apple Mail (2.1084) X-Gm-Message-State: ALoCoQninWUHm92egJMDW4ug3EBrCkYlO3yWGOR1Ws+E9H6NtFslwS3L0dzb0o1kAGYUv6PUwxOW Cc: freebsd-arm@freebsd.org, freebsd-arch@freebsd.org, Mark Tinguely , freebsd-mips@freebsd.org, Hans Petter Selasky Subject: Re: Partial cacheline flush problems on ARM and MIPS X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 26 Aug 2012 23:13:35 -0000 On Aug 26, 2012, at 12:25 PM, Ian Lepore wrote: > On Sun, 2012-08-26 at 13:05 -0500, Mark Tinguely wrote: >> I did a quick look at the drivers last summer. >>=20 >> Most drivers do the right thing and use memory allocated from >> bus_dmamem_alloc(). It is easy for us to give them a cache aligned >> buffer. >>=20 >> Some drivers use mbufs - 256 bytes which cache safe. >>=20 >> Some drivers directly or indirectly malloc() a buffer and then use it >> to dma - rather than try to fix them all, I was okay with making the >> smallest malloc() amount equal to the cache line size. It amounts to >> getting rid of the 16 byte allocation on some ARM architectures. The >> power of 2 allocator will then give us cache line safe allocation. >>=20 >> A few drivers take a small memory amount from the kernel stack and = dma >> to it <- broken driver. >>=20 >> The few drivers that use data from a structure and that memory is not >> cached aligned <- broken driver. >>=20 >=20 > I disagree about those last two points -- drivers that choose to use > stack memory or malloc'd memory as IO buffers are not broken. Stack DMA is bad policy, at best, and broken at worst. The reason is = because of alignment of the underlying unit. Since there's no way to = say that something is aligned to a given spot on the stack, you are = asking for random stack corruption. Also, malloced area is similarly problematic: There's no cache line = informing of the allocator, so you can wind up with an allocation of = memory that's corrupted due to cache effects. > Drivers > can do IO directly to/from userland buffers, do we say that an > application that calls read(2) and passes the address of a stack > variable is broken? Yes, if it is smaller than a cache line size, and not aligned to the = cache line. That's the point of the uio load variant. > In this regard, it's the busdma implementation that's broken, because = it > should bounce those IOs through a DMA-safe buffer. There's absolutely > no rule that I've ever heard of in FreeBSD that says IO can only take > place using memory allocated from busdma. That's partially true. Since BUSDMA grew up in the storage area, you = must allocate the memory from busdma, or it must be page aligned has = been the de-facto rule here. The mbuf and uio variants of load were = invented to cope with common cases of mbufs and user I/O to properly = flag things. How does busdma know that it is using memory that's not from its = allocator? > The rule is only that the > proper sequence of busdma operation must be called, and beyond that = it's > up to the busdma implementation to make it work. =20 No. Bouncing is needed due to poor alignment of the underlying device. = Not due to cache effects. There's a limited number of things that we support with busdma. = Arbitrary data from malloc that might be shared with the CPU isn't on = that list. > Our biggest problem, I think, is that we don't have a sufficient > definition of "the proper sequence of busdma operations." I disagree. The sequence has been known for a long time. > I don't think it will be very hard to make the arm and mips busdma > implementations work correctly. It won't even be too hard to make = them > fairly efficient at bouncing small IOs (my thinking is that we can = make > small bounces no more expensive than the current partial cacheline = flush > implementation which copies the data multiple times). Bouncing large = IO > will never be efficient, but the inefficiency will be a powerful > motivator to update drivers that do large IO to work better, such as > using buffers allocated from busdma. I don't think the cache line problem can be solved with bounce buffers. = Trying to accommodate broken drivers is what lead us to this spot. We = need to fix the broken drivers. If that's impossible, then the best we = can do is have the driver set a 'always bounce' flag in the tag it = creates and use that to always bounce for operations through that tag. Warner