From owner-freebsd-virtualization@freebsd.org Sun Jul 5 21:26:25 2015 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 701A498E84D for ; Sun, 5 Jul 2015 21:26:25 +0000 (UTC) (envelope-from neelnatu@gmail.com) Received: from mail-wi0-x229.google.com (mail-wi0-x229.google.com [IPv6:2a00:1450:400c:c05::229]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 0F3AE10B9; Sun, 5 Jul 2015 21:26:25 +0000 (UTC) (envelope-from neelnatu@gmail.com) Received: by wiwl6 with SMTP id l6so268373432wiw.0; Sun, 05 Jul 2015 14:26:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=2fraxoEn2UjD2xQ4lZ7YFnrZR65ShdpSBnU4ZcROcGg=; b=ZF4kJhA02OJ/2bCF0HTu70nGglPh0cX1iZn+8TM1vT23UsnPyfyxmFzToR5AWkwIKu ybOCM0/D/60YOuV6lHgFZVW+WNoEzfRV5ugc11J+a4R8sRabmEtiT9futeHJz71cyIzT yXI3hG6Br1lHCs1XTEuZdQHukgT2EJl/u4ZocIizNZYYoEqs0CZuUHoqpcfVwKuMqJwq VYyLclKU9fspbj7Ky35qXi7qYAusGasODFCweF7wkrhQzQOV0gYrkNF5NXSJZQ/BUbe1 1t/TfrTF1JBk4iHD01vukV0SoJR/O2LRM/TMiL8JtHEGsei6jAfmWiv3k9k4XAm9I/So AfEQ== MIME-Version: 1.0 X-Received: by 10.194.2.137 with SMTP id 9mr42104737wju.75.1436131583522; Sun, 05 Jul 2015 14:26:23 -0700 (PDT) Received: by 10.27.52.79 with HTTP; Sun, 5 Jul 2015 14:26:23 -0700 (PDT) In-Reply-To: References: Date: Sun, 5 Jul 2015 14:26:23 -0700 Message-ID: Subject: Re: bhyve - map user-space buffer in the VM From: Neel Natu To: Stefano Garzarella Cc: "freebsd-virtualization@freebsd.org" , Luigi Rizzo , Peter Grehan , Neel Natu , stefano@freebsd.org Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 05 Jul 2015 21:26:25 -0000 Hi Stefano, Sorry for the late reply. On Tue, Jun 30, 2015 at 2:57 AM, Stefano Garzarella wrote: > Hi all, > I'm working on bhyve to support ptnetmap (netmap passthrough) for my GSoC. > http://wiki.freebsd.org/SummerOfCode2015/ptnetmapOnBhyve > > I've already implemented it on QEMU/KVM and, to expose the netmap memory in > the VM, I used a PCI BAR and I mapped the netmap memory (the buffer returned > by mmap() on netmap fd in the user space) on the BAR in the VM through KVM > API. > > In bhyve I able to create a PCI BAR but I only found vm_map_pptdev_mmio() > API to map physical host page into the VM. I saw that this IOCTL is > implemented in the kernel module with sglist_append_phys() to populate a > scatter/gather list, than it is mapped in VM. > > I would like to add a new function in the API to map a user buffer in the > VM: > int vm_map_user_buf(struct vmctx *ctx, vm_paddr_t gpa, size_t len, void > *host_buf) > > I would implement it adding a new IOCTL and I would create, in the kernel > module, a new function like vmm_mmio_alloc() (sys/amd64/vmm/vmm_mem.c) but > using sglist_append_user() instead of sglist_append_phys(). > > What do you think about it? > There is another way to map an user-space buffer in the VM? > bhyve represents the guest address space as a vmspace that is independent from the bhyve process's vmspace. If a mapping is shared between the two address spaces then both mappings should point to the same vm_object. Also, the virtual machine lifetime can span the bhyve process lifetime. Therefore a shared mapping that points to the bhyve process's heap or data segment needs special handling. So, the general case of mapping an arbitrary bhyve user-space mapping into the guest address space is not trivial. Regarding 'vmm_mmio_alloc()' - this API maps PCI passthru device BARs into the guest address space. Note that it deals with both issues pointed above: 1. vmm.ko owns the passthru device so it can guarantee that the BAR addresses cannot change underneath the guest. 2. vm_cleanup() guarantees that the passthru mappings are removed from the guest address space before the next bhyve process starts running. The 'vmm_mmio_alloc()/vmm_mmio_free()' APIs are very narrow so should not be used a model for the general case. Having said that I think that for the specific case (netmap) your proposal might work assuming wired mappings of the netmap buffers. If the underlying netmap buffers are not physically contiguous then there will be a bit more work to do. I hope that allows you to make progress with the GSoC while we figure out a long-term viable solution. best Neel > Thanks, > Stefano >