Date: Mon, 13 Jun 2005 11:11:27 -0700 From: Julian Elischer <julian@elischer.org> To: Apache Xie <apachexm@hotmail.com> Cc: freebsd-hackers@freebsd.org Subject: Re: contigmalloc() and mmap() Message-ID: <42ADCC4F.4090201@elischer.org> In-Reply-To: <BAY19-F33CED9E68F54579EE49ACCC4F00@phx.gbl> References: <BAY19-F33CED9E68F54579EE49ACCC4F00@phx.gbl>
next in thread | previous in thread | raw e-mail | index | archive | help
2nd try to answer this.. Apache Xie wrote: > I have some experiences in writing Linux device driver, > but I am new to FreeBSD kernel, although from the first > glimpse, there seems no big differences between the kernel > operations a char device driver can use, but I met some > problems when the driver is running in FreeBSD. > > Our device is an experimental communication card, which can > do remote DMA between two cards, which means the card in one > node can DMA memory data to network, and when data are transfered > to the card on another node, it will be DMAed to memory too. > > > Because the card can only use contiguous physical memory for > DMA operations, so data in user process will be copied to an > contiguous memory buffer, then DMA will tranfer data in this > buffer, and this buffer is allocated by driver using __get_free_pages() > in Linux. The buffer is mmaped to user process space, so user > process can do the copy directly in user space, it can directly > orgnize data in this mmaped memory too. ok, so far so good.. machine to machine DMA capability with contiguous buffers.. nothing too complicated so far.. I once did a simiar system on BSD4.3 in 1992. > > When I am porting my driver to FreeBSD, I use /dev/bktr driver as the > example, seems easily, just using contigmalloc() to allocate the > buffer, and in driver's _mmap() function, return the physical > address for each page to be mmaped. > > The problem is, in Linux, I allocate buffer in driver's ioctl() > function, and free it in a timer function, many processes may use the > driver > at the same time, each process use a different kernel buffer, when > the process first use the driver, it calls __get_free_pages() to allocate > kernel buffer, and when it exit, it trigger timer function, the timer > function will can free_pages() to free the buffer, so these two kernel > interfaces will be called frequently, but this usage pattern works > correctly in Linux. I'm unsure about this usage of the timer (callout(?) ) how does the timer know which buffer pages to remove? and if each userspace process does an ioctl to allocate a different buffer, are the new pages also visible to other processes? in FreeBSD your driver can register with at_exit() in 4.x and with eventhandler_register() in 5.x or 6.x to have stuff done when the process exits. (there are also at_fork and at_exec handlers too.) see man at_exit (etc.) in 4.x and man 9 EVENTHANDLER in 5.x and 6.x You could also make a hash table on the PID so that you keep different information available for different processes and look them up using curproc->p_pid (for 4.x) and curthhread->td_proc->p_pid for 5.x and 6.x to look up the Using these primatives you should be able to simulate what you need I believe. > In FreeBSD, the driver works in the same pattern, but when a user process > mmap driver's buffer (allocated by contigmalloc()) and is killed, then > when > another process mmap the same buffer again, sometimes it cannot get > correct > data from the mmaped pages, which means the user space virtual aderess > may > not point to the correct physical page of driver's buffer, sometimes > the OS > even panic with some information such as "Trap 12, Page not present" etc. sounds like you are getting the buffers confused.. are you doing correc treference counting on the buffers? > > I browsed kernel tree, I found those drivers which use contigmalloc() and > contigfree() always call these two kernel interfaces in _attach() and > _detach(), but in my driver, I call contigmalloc() in ioctl(), and call > contigfree() in a callout function which is set by callout_reset(). why by callout_reset()? > What I want to know is if contigmalloc() and contigfree() can only be > used > under some conditions? not that I know of.. > > And recently, I modified my driver, I allocated a big chunk of contiguous > physical memory using contigmalloc() in the driver's _attach() function, > and I use a simple first-fit algorithm to manage this memory myself, > which > mean in ioctl() I use my allocate/deallocate functions instead of > contigmalloc(), > in _detach() function contigfree() is called to free the big chunk of > memory, > no panic again, but sometimes, process cannot get the correct data from > the mmaped memory. I don't know why. it does sound a bit like you are not keeping the information separated between processes enough. > > Any help is welcomed. > > Thanks. > > _________________________________________________________________ > Express yourself instantly with MSN Messenger! Download today it's > FREE! http://messenger.msn.click-url.com/go/onm00200471ave/direct/01/ > > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to > "freebsd-hackers-unsubscribe@freebsd.org"
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?42ADCC4F.4090201>