From owner-freebsd-current@freebsd.org Tue Apr 5 17:02:12 2016 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B81D0B035BB for ; Tue, 5 Apr 2016 17:02:12 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from bigwig.baldwin.cx (bigwig.baldwin.cx [IPv6:2001:470:1f11:75::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 988C91D46 for ; Tue, 5 Apr 2016 17:02:12 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from ralph.baldwin.cx (c-73-231-226-104.hsd1.ca.comcast.net [73.231.226.104]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id A4982B93C; Tue, 5 Apr 2016 13:02:11 -0400 (EDT) From: John Baldwin To: Konstantin Belousov Cc: freebsd-current@freebsd.org, Ryan Stone Subject: Re: accessing a PCIe register from userspace through kmem or other ways ? Date: Tue, 05 Apr 2016 10:02:07 -0700 Message-ID: <8615362.BAVEr0Gv7s@ralph.baldwin.cx> User-Agent: KMail/4.14.3 (FreeBSD/10.2-STABLE; KDE/4.14.3; amd64; ; ) In-Reply-To: <20160405162129.GG1741@kib.kiev.ua> References: <2874767.SJKfSzhn1q@ralph.baldwin.cx> <20160405162129.GG1741@kib.kiev.ua> MIME-Version: 1.0 Content-Transfer-Encoding: 7Bit Content-Type: text/plain; charset="us-ascii" X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Tue, 05 Apr 2016 13:02:11 -0400 (EDT) X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Apr 2016 17:02:12 -0000 On Tuesday, April 05, 2016 07:21:29 PM Konstantin Belousov wrote: > On Tue, Apr 05, 2016 at 08:55:54AM -0700, John Baldwin wrote: > > Mostly I do not have experience with MGTDEVICE, though I was planning to > > look at it as a way to implement this. Two things though: 1) there may > > not be a cdev to associate with, and 2) I know of at least one device driver > > that would use this in addition to using this for a general "map this BAR" > > ioctl on /dev/pci. > So /dev/pci is the natural cdev to place the functionality. > An ioctl on /dev/pci may mmap BAR and return the base address. > > > There are other cases in the past where I used OBJT_SG > > but would have preferred to use a variant that used managed pages so that > > I could invidate any existing mappings. In particular what I want to do > > is invalidate an object so that any future uses fail. > > > > Alternatively, it might be nice to hook a destructor call into a VM object > > so that I could know when the object is no longer in use (knowing that all > > its mappings have been destroyed). When using OBJT_SG objects as aliases > > for other things (memory allocated via contigmalloc or bus_dma or for > > resources like PCI BARs), I could keep a reference count on the original > > "thing" that I increment when creating an OBJT_SG object to return from > > something like d_mmap_single() or the /dev/pci ioctl and drop the reference > > count in the destructor hook for that object. > This is in essence how GEM objects + MGTDEVICE mappings work for i915. > > The only bottleneck in the API arrangement is that d_mmap_single() only > gets the offset as the identifying data to construct the mapping. > For /dev/pci, the offset parameter would need to encode d:b:s:f and BAR > index. For the ioctl I planned to either 1) call vm_mmap_object() or the like directly and return the virtual address to the user, or 2) return the mmap offset to use from the ioctl that the user would then supply to mmap() and d_mmap_single would use to find the object created by the ioctl. 1) is probably simpler and is more what I was leaning towards. Still, I want to be able to handle invalidations either by pinning the BAR while the object is mapped, or being able to invalidate the object. Given that you can eject a hotplug PCI device, I think explicit invalidation is the better route in this case. I would create a VM object for each BAR on the first mmap request and save a reference to it in the PCI bus ivars. If the BAR is ever cleared I would be able to find the object and invalidate it ensuring any programs that then tried to access it would get a page fault instead of accessing some other random thing. -- John Baldwin