Date: Wed, 11 Jan 2017 19:09:21 -0800 From: Peter Grehan <grehan@freebsd.org> To: soralx@cydem.org Cc: freebsd-virtualization@freebsd.org Subject: Re: Issues with GTX960 on CentOS7 using bhyve PCI passthru (FreeBSD 11-RC2) Message-ID: <839b4e54-fd7a-20fd-630f-f8aaee0c79ee@freebsd.org> In-Reply-To: <20170111184810.5e9fb07f@mscad14> References: <20170110003332.7cf8ba15@mscad14> <0de7e0fe-5680-b1be-bd57-6bf446c2fd38@talk2dom.com> <0c927784-3e3f-7946-fba9-c25001f4156c@talk2dom.com> <20170110180117.7f246b5a@mscad14> <20170111014544.70670784@mscad14> <93196ea2-5439-49ff-54fd-7b7273bdec85@freebsd.org> <20170111184810.5e9fb07f@mscad14>
next in thread | previous in thread | raw e-mail | index | archive | help
Hi, >> That's correct - it's a bug in bhyve. > > Baking a proper fix will be more complicated by the fact that PCIe > cards themselves may have limitations. For example, most nVidia GPUs > have 40 bits DMA addressing capability, some 39, an a few (still > modern) ones -- just 37 [ref. nVidia "README" in the driver package]. That's a different issue - it's unlikely, if not impossible, to configure bhyve with enough RAM to hit 37 bits worth where that would become a problem. No need to worry about that. >> PCI passthru doesn't allow the BAR values to be modified (this could >> be changed, but it's a lot of work for little gain). > > Removing another signature of detecting virtualization and increasing > compatibility would be negligible gain? Just asking... There are lots of BIOS/UEFI implementations out there that have the same restriction. In general, there should be no need for a guest to reprogram device BARs. After changing the 64-bit BAR base address, did you still need the pci=nocrs option for Linux ? I'd hope this would be no longer necessary. >>> But: >>> # ./nvidia-smi >>> No devices were found >>> dmesg: >>> [ 173.498953] NVRM: RmInitAdapter failed! (0x53:0x3:1856) >>> [ 173.499115] NVRM: rm_init_adapter failed for device bearing >>> minor number 0 >> >> Looks like you're getting close :) > > Hmm, I'm not seeing myself getting much closer here. Do you know > something I don't? ;) I really hope bhyve developers can spare a > bit of time on getting GPU passthrough to work... I know nothing > about these things, and where I waste half a day messing around, > the problem could be fixed in half an hour by someone who knows. The problem is the knowledge set of graphics/GPU knowhow and equipment access, and bhyve/PCI programming, are disjoint. The time I've spent on it has been the inverse, where I feel that I've spent a half-day doing things that anyone who knew about graphics could get done in a half-hour :) For these type of issues, joint work is best to leverage the knowledge of both sides. From my point-of-view, the work you've done has been very helpful. later, Peter.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?839b4e54-fd7a-20fd-630f-f8aaee0c79ee>