Date: Thu, 13 Dec 2001 18:09:54 +0100 (CET) From: =?ISO-8859-1?Q?G=E9rard_Roudier?= <groudier@free.fr> To: Greg Johnson <gjohnson@research.canon.com.au> Cc: <freebsd-hackers@FreeBSD.ORG> Subject: Re: Bus master DMA problems Message-ID: <20011213175415.R1979-100000@gerard> In-Reply-To: <20011213041426.77707B7331@brixi.research.canon.com.au>
next in thread | previous in thread | raw e-mail | index | archive | help
There are a couple of rules in PCI you must have in mind when synchronization between PCI devices is needed. 1) Interrupts are not synchronization events. They just send attention to the device driver (acts as SIGIO, for example). Some bridge may flush posted buffers on interrupt, but since interrupt can be shared, you must not rely on such mechanism. 2) After a write transaction has completed to some target, you must read some data from this target if you want posted buffers to be flushed. (perform a dummy read to some IO register if needed). 3) Prior to looking at data that just have been DMAed, perform a read barrier in order to flush any data prefetched by the CPU. Otherwise you may see stale data. 4) Write barriers must be used to enforce ordering of writes when the system may reorder them. But this does not apply to IA32 architecture that guarantees ordering of write as seen from other agents. The dmamap_* are intended to deal with systems specific issues, system bridges, CPU, etc.... They are not enough to ensure synchronisation will be what your software expects. Even in the hypothesis of them being enough for a given system, a PCI board that incorporates a PCI bridge will require at least rule #2 not to be missed. G=E9rard. On Thu, 13 Dec 2001, Greg Johnson wrote: > Hello, > > I am working on a FreeBSD device driver for a custom > PCI bus device developed by my company. The PCI card > has large amounts on memory on it and we are using > bus master DMA to get data in and out of memory. > > The proceedure that I am using is as follows: > > A.1. The DMA controller on our device is set up > for the DMA transfer of memory. > > A.2. asleep() is called to let the OS know that we > are about to wait on an event. > > A.3. An enable bit is set on the device to start the > transfer. > > A.4. await() is called to get the current process to > sleep and wait for the event. > > A.5. The ISR, upon recipt of an interrupt that indicates > that the DMA transfer is complete, calls wakeup() > on the ident for the process sleeping, waiting for the > DMA transfer. > > A.6. await() returns. > > In addition, the memory buffers on the PC host side are > initially allocated by a user process, and locked down > (wired) buy the device driver so that they a present > in physical memory. > > A bus DMA Tag is created once on driver load that > describes the DMA transfers. > > On a call to read or write: > > B.1. Memory is locked. > > B.2. bus_dmamap_sync() is called (PRE*) > > B.3. bus_dmamap_load() is called > > B.4. bus_dmamap sync() is called (POST*) > > B.5. Memory is unlocked. > > Now, the problem I am experiencing is that memory does not > appear to be DMAed correctly. When using the interrupt driver > method above, sometimes there are errors in the data written > to the device, and sometimes in the data read back from the > device. > > When using a polling approach and dissabling the DMA interrupts, > starting the transfer and polling for when the transfer is complete, > the system works correctly. > > Also, when using the interrupt approach described, if I insert a > large delay of about 100us (50us does not work) in the ISR at the > point when the ISR confirms that the interrupt is the DMA done > interrupt, then everything works correctly. > > The problem sounds a lot like out PCI device is not completing > the DMA transfer when the interrupt is raised, but with the polling > process, it only seems to take a few (2-8)us before the DMA done > interrupt is asserted and the call returns, and in this case > everything works. > > So if anyong out there in FreeBSD guru land has any ideas as > to what might be wrong, or things I might of missed in seeting > up stuff, I would really appreciate their input. > > Our systems are x86 (PIIIs and P4s) running FreeBSD 4.4-stable. > > Thanks. > > Greg. > > -- > +------------------------------------------------------+ > | Do you want to know more? www.geocities.com/worfsom/ | > | ..ooOO Greg Johnson OOoo.. | > | HW/SW Engineer gjohnson@research.canon.COM.Au | > | Canon Information Systems Research Australia (CISRA) | > | 1 Thomas Holt Dr., North Ryde, NSW, 2113, Australia | > | "I FLEXed my BISON and it went YACC!" - me. | > +------------------------------------------------------+ > > > To Unsubscribe: send mail to majordomo@FreeBSD.org > with "unsubscribe freebsd-hackers" in the body of the message To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-hackers" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20011213175415.R1979-100000>