From owner-freebsd-current@FreeBSD.ORG  Tue Jan 10 22:24:01 2012
Return-Path: <owner-freebsd-current@FreeBSD.ORG>
Delivered-To: freebsd-current@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id 80A42106567A;
	Tue, 10 Jan 2012 22:24:01 +0000 (UTC)
	(envelope-from luigi@onelab2.iet.unipi.it)
Received: from onelab2.iet.unipi.it (onelab2.iet.unipi.it [131.114.59.238])
	by mx1.freebsd.org (Postfix) with ESMTP id 400A98FC20;
	Tue, 10 Jan 2012 22:24:00 +0000 (UTC)
Received: by onelab2.iet.unipi.it (Postfix, from userid 275)
	id 751A37300A; Tue, 10 Jan 2012 23:41:00 +0100 (CET)
Date: Tue, 10 Jan 2012 23:41:00 +0100
From: Luigi Rizzo <rizzo@iet.unipi.it>
To: Adrian Chadd <adrian@freebsd.org>
Message-ID: <20120110224100.GB93082@onelab2.iet.unipi.it>
References: <20120110213719.GA92799@onelab2.iet.unipi.it>
	<CAJ-VmomdQ5ZWBf_h1xJhppO8WsinvK7RJiDSgDrYKpo+J8eGYQ@mail.gmail.com>
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <CAJ-VmomdQ5ZWBf_h1xJhppO8WsinvK7RJiDSgDrYKpo+J8eGYQ@mail.gmail.com>
User-Agent: Mutt/1.4.2.3i
Cc: FreeBSD current <freebsd-current@freebsd.org>
Subject: Re: memory barriers in bus_dmamap_sync() ?
X-BeenThere: freebsd-current@freebsd.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: Discussions about the use of FreeBSD-current
	<freebsd-current.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-current>, 
	<mailto:freebsd-current-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-current>
List-Post: <mailto:freebsd-current@freebsd.org>
List-Help: <mailto:freebsd-current-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-current>,
	<mailto:freebsd-current-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Tue, 10 Jan 2012 22:24:01 -0000

On Tue, Jan 10, 2012 at 01:52:49PM -0800, Adrian Chadd wrote:
> On 10 January 2012 13:37, Luigi Rizzo <rizzo@iet.unipi.it> wrote:
> > I was glancing through manpages and implementations of bus_dma(9)
> > and i am a bit unclear on what this API (in particular, bus_dmamap_sync() )
> > does in terms of memory barriers.
> >
> > I see that the x86/amd64 and ia64 code only does the bounce buffers.
> > The mips seems to do some coherency-related calls.
> >
> > How do we guarantee, say, that a recently built packet is
> > to memory before issuing the tx command to the NIC ?
> 
> The drivers should be good examples of doing the right thing. You just
> do pre-map and post-map calls as appropriate.
> 
> Some devices don't bother with this on register accesses and this is a
> bug. (eg, ath/ath_hal.) Others (eg iwn) do explicit flushes where
> needed.

so you are saying that drivers are correct unless they are buggy :)

Anyways... i see that some drivers use wmb() and rmb() and redefine
their own version, usually based on lfence/sfence even on i386

	#define rmb()	__asm volatile("lfence" ::: "memory")
	#define wmb()	__asm volatile("sfence" ::: "memory")

whereas the standard definitions are slightly different, e.g.
sys/i386/include/atomic.h:

    #define      rmb()   __asm __volatile("lock; addl $0,(%%esp)" : : : "memory")
    #define      wmb()   __asm __volatile("lock; addl $0,(%%esp)" : : : "memory")

and our bus_space API in sys/x86/include/bus.h is a bit unclear to
me (other than the fact that having 4 unused arguments don't really
encourage its use...)


    static __inline void
    bus_space_barrier(bus_space_tag_t tag __unused, bus_space_handle_t bsh __unused,
		      bus_size_t offset __unused, bus_size_t len __unused, int flags)
    {
    #ifdef __GNUCLIKE_ASM
	    if (flags & BUS_SPACE_BARRIER_READ)
    #ifdef __amd64__
		    __asm __volatile("lock; addl $0,0(%%rsp)" : : : "memory");
    #else
		    __asm __volatile("lock; addl $0,0(%%esp)" : : : "memory");
    #endif
	    else
		    __asm __volatile("" : : : "memory");
    #endif
    }

cheers
luigi