From owner-freebsd-virtualization@FreeBSD.ORG Tue Jan 20 16:37:15 2015 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id EAEF9970; Tue, 20 Jan 2015 16:37:15 +0000 (UTC) Received: from smtp.digiware.nl (smtp.digiware.nl [31.223.170.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id A4443929; Tue, 20 Jan 2015 16:37:15 +0000 (UTC) Received: from rack1.digiware.nl (unknown [127.0.0.1]) by smtp.digiware.nl (Postfix) with ESMTP id 039B916A405; Tue, 20 Jan 2015 17:37:12 +0100 (CET) X-Virus-Scanned: amavisd-new at digiware.nl Received: from smtp.digiware.nl ([127.0.0.1]) by rack1.digiware.nl (rack1.digiware.nl [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id NaLQ3tsbgtDT; Tue, 20 Jan 2015 17:37:10 +0100 (CET) Received: from [IPv6:2001:4cb8:3:1:6455:2a07:d7dc:ef6e] (unknown [IPv6:2001:4cb8:3:1:6455:2a07:d7dc:ef6e]) by smtp.digiware.nl (Postfix) with ESMTP id D680A16A403; Tue, 20 Jan 2015 17:37:10 +0100 (CET) Message-ID: <54BE8434.3070901@digiware.nl> Date: Tue, 20 Jan 2015 17:37:08 +0100 From: Willem Jan Withagen Organization: Digiware Management b.v. User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:31.0) Gecko/20100101 Thunderbird/31.4.0 MIME-Version: 1.0 To: Peter Grehan Subject: How to connect ssh and /dev/nmdm devices Was:( Re: Native Linux guest in Bhyve (no grub2-bhyve) status? ) References: <5448305F.8010307@freebsd.org> <54483712.8050007@freebsd.org> <54A9CAFB.1090902@digiware.nl> <54A9CC90.2020103@freebsd.org> <54AA4E32.1080303@digiware.nl> In-Reply-To: <54AA4E32.1080303@digiware.nl> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit Cc: Conrad Rad , "freebsd-virtualization@freebsd.org" X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Jan 2015 16:37:16 -0000 On 2015-01-05 9:41, Willem Jan Withagen wrote: > On 2015-01-05 0:28, Peter Grehan wrote: >> Hi Willem, >> >>> Would it it be possible in the meantime to enhance grub2-bhyve with the >>> same possibility as bhyve itself has for the console? >>> >>> So I can redirect grub screen access to /dev/nmdm12041, and one can even >>> use this channel to see the grubscreen during rebooting. >>> >>> Or if it is not that hard, give some pointers to write it myself. >> >> Conrad contributed this to grub-bhyve with >> https://github.com/grehan-freebsd/grub2-bhyve/pull/2 >> >> It's available in v0.30 which has been in ports for a while now: >> >> https://github.com/grehan-freebsd/grub2-bhyve/releases/tag/v0.30 > > Right, > > Did miss that, I guess. > I'll upgrade my bhyve-grub. Yup, works really nice. Now in a continuance from this... What is the easiest way to "propagate" the full-duplex tty stream from a SSH-login to a /dev/nmdmXXXXA. This will give easy access to the console screens. one would type: ssh -p XXXX user@management-ip and end up at the console on /dev/nmdmxxxxA Currently I'm using tip for this, but that also requires hacking new VMs into /etc/remote. So a very basic TIP, without the serial/ACU/speed stuff, would really be useful. Preferably all signals are sent thru as well. And if this might not be 100% secure, it would require an ssh-jailed setup. A bit like sftp. hints and or suggestion welcome. --WjW --WjW From owner-freebsd-virtualization@FreeBSD.ORG Tue Jan 20 17:48:32 2015 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id DBFBA77D; Tue, 20 Jan 2015 17:48:32 +0000 (UTC) Received: from family.redbarn.org (family.redbarn.org [IPv6:2001:559:8000:cd::5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id B49AAEF; Tue, 20 Jan 2015 17:48:32 +0000 (UTC) Received: from [IPv6:2001:559:8000:cb:98f0:bf77:6e47:b4fd] (unknown [IPv6:2001:559:8000:cb:98f0:bf77:6e47:b4fd]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by family.redbarn.org (Postfix) with ESMTPSA id 4C6E113B47; Tue, 20 Jan 2015 17:48:32 +0000 (UTC) Message-ID: <54BE94EC.1080806@redbarn.org> Date: Tue, 20 Jan 2015 09:48:28 -0800 From: Paul Vixie User-Agent: Postbox 3.0.11 (Windows/20140602) MIME-Version: 1.0 To: Willem Jan Withagen Subject: Re: How to connect ssh and /dev/nmdm devices Was:( Re: Native Linux guest in Bhyve (no grub2-bhyve) status? ) References: <5448305F.8010307@freebsd.org> <54483712.8050007@freebsd.org> <54A9CAFB.1090902@digiware.nl> <54A9CC90.2020103@freebsd.org> <54AA4E32.1080303@digiware.nl> <54BE8434.3070901@digiware.nl> In-Reply-To: <54BE8434.3070901@digiware.nl> X-Enigmail-Version: 1.2.3 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: "freebsd-virtualization@freebsd.org" , Conrad Rad X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Jan 2015 17:48:33 -0000 > Willem Jan Withagen > Tuesday, January 20, 2015 8:37 AM > > > > What is the easiest way to "propagate" the full-duplex tty stream > from a SSH-login to a /dev/nmdmXXXXA. somewhat expectedly, i use for this. with a restricted shell, i can offer console access to guests via ssh. "cu" also works, but lacks rtty's background logging. -- Paul Vixie From owner-freebsd-virtualization@FreeBSD.ORG Tue Jan 20 19:00:46 2015 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id F024A507; Tue, 20 Jan 2015 19:00:45 +0000 (UTC) Received: from gold.funkthat.com (gate2.funkthat.com [208.87.223.18]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client CN "gold.funkthat.com", Issuer "gold.funkthat.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id C624ABAF; Tue, 20 Jan 2015 19:00:45 +0000 (UTC) Received: from gold.funkthat.com (localhost [127.0.0.1]) by gold.funkthat.com (8.14.5/8.14.5) with ESMTP id t0KJ0dWu061948 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 20 Jan 2015 11:00:39 -0800 (PST) (envelope-from jmg@gold.funkthat.com) Received: (from jmg@localhost) by gold.funkthat.com (8.14.5/8.14.5/Submit) id t0KJ0cUL061946; Tue, 20 Jan 2015 11:00:38 -0800 (PST) (envelope-from jmg) Date: Tue, 20 Jan 2015 11:00:38 -0800 From: John-Mark Gurney To: Willem Jan Withagen Subject: Re: How to connect ssh and /dev/nmdm devices Was:( Re: Native Linux guest in Bhyve (no grub2-bhyve) status? ) Message-ID: <20150120190038.GR1949@funkthat.com> References: <5448305F.8010307@freebsd.org> <54483712.8050007@freebsd.org> <54A9CAFB.1090902@digiware.nl> <54A9CC90.2020103@freebsd.org> <54AA4E32.1080303@digiware.nl> <54BE8434.3070901@digiware.nl> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <54BE8434.3070901@digiware.nl> X-Operating-System: FreeBSD 9.1-PRERELEASE amd64 X-PGP-Fingerprint: 54BA 873B 6515 3F10 9E88 9322 9CB1 8F74 6D3F A396 X-Files: The truth is out there X-URL: http://resnet.uoregon.edu/~gurney_j/ X-Resume: http://resnet.uoregon.edu/~gurney_j/resume.html X-TipJar: bitcoin:13Qmb6AeTgQecazTWph4XasEsP7nGRbAPE X-to-the-FBI-CIA-and-NSA: HI! HOW YA DOIN? can i haz chizburger? User-Agent: Mutt/1.5.21 (2010-09-15) X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.7 (gold.funkthat.com [127.0.0.1]); Tue, 20 Jan 2015 11:00:39 -0800 (PST) Cc: "freebsd-virtualization@freebsd.org" , Conrad Rad X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Jan 2015 19:00:46 -0000 Willem Jan Withagen wrote this message on Tue, Jan 20, 2015 at 17:37 +0100: > On 2015-01-05 9:41, Willem Jan Withagen wrote: > > On 2015-01-05 0:28, Peter Grehan wrote: > >> Hi Willem, > >> > >>> Would it it be possible in the meantime to enhance grub2-bhyve with the > >>> same possibility as bhyve itself has for the console? > >>> > >>> So I can redirect grub screen access to /dev/nmdm12041, and one can even > >>> use this channel to see the grubscreen during rebooting. > >>> > >>> Or if it is not that hard, give some pointers to write it myself. > >> > >> Conrad contributed this to grub-bhyve with > >> https://github.com/grehan-freebsd/grub2-bhyve/pull/2 > >> > >> It's available in v0.30 which has been in ports for a while now: > >> > >> https://github.com/grehan-freebsd/grub2-bhyve/releases/tag/v0.30 > > > > Right, > > > > Did miss that, I guess. > > I'll upgrade my bhyve-grub. > > Yup, works really nice. > > Now in a continuance from this... > > What is the easiest way to "propagate" the full-duplex tty stream from > a SSH-login to a /dev/nmdmXXXXA. > > This will give easy access to the console screens. > > one would type: > ssh -p XXXX user@management-ip > and end up at the console on /dev/nmdmxxxxA > > Currently I'm using tip for this, but that also requires hacking new VMs > into /etc/remote. > > So a very basic TIP, without the serial/ACU/speed stuff, would really be > useful. Preferably all signals are sent thru as well. > And if this might not be 100% secure, it would require an ssh-jailed > setup. A bit like sftp. > > hints and or suggestion welcome. If you want this, look at RFC2217 compatible programs... RFC2217 extends telnet to provide changing things like baud rates and parity inline... The Python module PySerial includes both a client: http://pyserial.sourceforge.net/examples.html#miniterm and a server: http://pyserial.sourceforge.net/examples.html#single-port-tcp-ip-serial-bridge-rfc-2217 Looks like ports has sredird that does the same for a server... So, you could use a server attached the nmdm plus ssh port tunneling to provide access... I've been tempted to write a program using cuse that would present a modem device that could be used w/ tip or cu and connect to an RFC2217 server on the back end giving you complete remove access to all settings of the serial port, but haven't gotten around to doing that yet... -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not." From owner-freebsd-virtualization@FreeBSD.ORG Tue Jan 20 20:44:57 2015 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 03D91870; Tue, 20 Jan 2015 20:44:57 +0000 (UTC) Received: from mail-lb0-x229.google.com (mail-lb0-x229.google.com [IPv6:2a00:1450:4010:c04::229]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 7D83E9B0; Tue, 20 Jan 2015 20:44:56 +0000 (UTC) Received: by mail-lb0-f169.google.com with SMTP id f15so15116754lbj.0; Tue, 20 Jan 2015 12:44:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:date:message-id:subject:from:to:cc:content-type; bh=DYzUZVO9xODkHRmnW4tipedQfZQbwCPK++6s8XtRojE=; b=oBMQL4ikzZtenGLUqB4jqpv7iXFJ36psLxjkoFGXiSTlArlN16czfodb49NDW+spCk 0HHcweb1lNXuJu42vkAOLYSoYk7FSwar+ycPlvPwLJTtt7kmffkSrjHsv2ILbEEc+VLB kk0azxU1VAYat/6uTz+XwdPg4IPB6KbjUQaDjlWRC84Bvt1WJmh+lZzHH5TWrtBuvBYr eH6ILQMA835w4IE/s7Ju0hHQ6ED+Pc9OFu/og2gX5ImnpDvj3JKaDu5oxncevi7z8Zkf B/bEOAq9BzuLXBqI7w80x4D7Pyloh8dTwVyRfLihyZSzV8hSSequ8PDq+OXmW4WjyVPO OAnA== MIME-Version: 1.0 X-Received: by 10.152.23.98 with SMTP id l2mr40263025laf.46.1421786694550; Tue, 20 Jan 2015 12:44:54 -0800 (PST) Sender: crodr001@gmail.com Received: by 10.112.129.3 with HTTP; Tue, 20 Jan 2015 12:44:54 -0800 (PST) Date: Tue, 20 Jan 2015 12:44:54 -0800 X-Google-Sender-Auth: EBlS5d_GL5y6tezGI2DM868NoEg Message-ID: Subject: Re: How to connect ssh and /dev/nmdm devices From: Craig Rodrigues To: Willem Jan Withagen Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: "freebsd-virtualization@freebsd.org" , Conrad Rad X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Jan 2015 20:44:57 -0000 On Tue, Jan 20, 2015 at 8:37 AM, Willem Jan Withagen wrote: > > Now in a continuance from this... > > What is the easiest way to "propagate" the full-duplex tty stream from a > SSH-login to a /dev/nmdmXXXXA. > I've used http://www.freshports.org/comms/conserver-com/ to set up a "terminal server" that connects to /dev/nmdm devices. -- Craig From owner-freebsd-virtualization@FreeBSD.ORG Thu Jan 22 18:11:10 2015 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 637E26ED; Thu, 22 Jan 2015 18:11:10 +0000 (UTC) Received: from smtp.digiware.nl (smtp.digiware.nl [31.223.170.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 083AAE66; Thu, 22 Jan 2015 18:11:09 +0000 (UTC) Received: from rack1.digiware.nl (unknown [127.0.0.1]) by smtp.digiware.nl (Postfix) with ESMTP id 4A6AD16A404; Thu, 22 Jan 2015 19:11:01 +0100 (CET) X-Virus-Scanned: amavisd-new at digiware.nl Received: from smtp.digiware.nl ([127.0.0.1]) by rack1.digiware.nl (rack1.digiware.nl [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id L1G_mh3ZkbIy; Thu, 22 Jan 2015 19:10:59 +0100 (CET) Received: from [IPv6:2001:4cb8:3:1:6455:2a07:d7dc:ef6e] (unknown [IPv6:2001:4cb8:3:1:6455:2a07:d7dc:ef6e]) by smtp.digiware.nl (Postfix) with ESMTP id 71ACB16A401; Thu, 22 Jan 2015 19:10:59 +0100 (CET) Message-ID: <54C13D2D.40609@digiware.nl> Date: Thu, 22 Jan 2015 19:10:53 +0100 From: Willem Jan Withagen Organization: Digiware Management b.v. User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:31.0) Gecko/20100101 Thunderbird/31.4.0 MIME-Version: 1.0 To: "freebsd-virtualization@freebsd.org" Subject: Re: How to connect ssh and /dev/nmdm devices References: <5448305F.8010307@freebsd.org> <54483712.8050007@freebsd.org> <54A9CAFB.1090902@digiware.nl> <54A9CC90.2020103@freebsd.org> <54AA4E32.1080303@digiware.nl> <54BE8434.3070901@digiware.nl> <54BE94EC.1080806@redbarn.org> In-Reply-To: <54BE94EC.1080806@redbarn.org> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit Cc: Conrad Rad X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 22 Jan 2015 18:11:10 -0000 On 2015-01-20 18:48, Paul Vixie wrote: > > >> Willem Jan Withagen >> Tuesday, January 20, 2015 8:37 AM >> >> >> >> What is the easiest way to "propagate" the full-duplex tty stream >> from a SSH-login to a /dev/nmdmXXXXA. > > somewhat expectedly, i use > for this. with a restricted shell, i can offer console access to guests > via ssh. "cu" also works, but lacks rtty's background logging. HI, I got several suggestions, and in the end I went for this one. Why? The code base is small and simple it does exactly what I required, no more, no less. Getting it to work requires a bit of careful configuration but one you've done that you end up with SSH users which can do multiple logins and all see the same console. Input and things all the same. So no exclusion because you've left your terminal open at work and went home. Nifty stuff. Basic install: 1) pkg install rtty. This populates the /usr/local/rtty tree. And that is where just almost everything happens. 2) In /usr/local/rtty/dev create links to the nmdm-devices you'd like to connect to the console: root@bhyve00:/usr/local/rtty # ls -l dev total 1 lrwxr-xr-x 1 root wheel 14 Jan 22 11:13 bh1101A -> /dev/nmdm1101B 3) Create a "shell" to connect to this console: root@bhyve00:/usr/local/rtty # cat bin/rtty-shell #!/bin/sh exec /usr/local/rtty/bin/console "$LOGNAME" # exec /usr/bin/cu -t -l /dev/nmdm1101A # exec /usr/bin/tip vm1101 exit And make it executable: root@bhyve00:/usr/local/rtty # ls -l bin/rtty-shell -rwxr-xr-x 1 root wheel 127 Jan 22 11:15 bin/rtty-shell As you can see you can use other commands as well, but both tip and cu are all about exclusive access. Trick here is that LOGNAME is what the user is using for login, and that is also the name of the device in ../dev to which he/she is connected. 4) Create the user in /etc/passwd: root@bhyve00:/usr/local/rtty # grep bh1101A /etc/passwd bh1101A:*:10001:10001:Bhyve dome remote access user:/home/bhyvedemo:/usr/local/rtty/bin/rtty-shell And put him in /etc/group to make things complete: root@bhyve00:/usr/local/rtty # grep bh1101A /etc/group bh1101A:*:10001: Group is needed for the rights on the socket-device in step 6. 5) Which also requires an entry in /etc/shells to work: root@bhyve00:/usr/local/rtty # grep rtty /etc/shells /usr/local/rtty/bin/rtty-shell 6) Having done all this the we need to make sure that the accessrights on the linked device are set correctly: root@bhyve00:/usr/local/rtty # cat owner/bh1101A.sock root:bh1101A This will prevent another user from hijacking a console which is not his, in case of access by other means. 7) Last ting to do is to start the server deamon that does the multiplexing/sharing stuff: /usr/sbin/daemon -cf /usr/local/rtty/bin/Startup Which you could dump into /etc/rc.local, or the more brave would write a rc.d compliant startup script. 8) Test: root@bhyve00:/usr/local/rtty # ssh bh1101A@localhost Password for bh1101A@bhyve00: Last login: Wed Jan 21 12:42:37 2015 from mail.ip6.digiware.nl FreeBSD 11.0-CURRENT (GENERIC) #19 r277452: Wed Jan 21 08:01:54 UTC 2015 Welcome to FreeBSD! connected (use (CR)~? for minimal help; also (CR)~q? and (CR)~s?) [authorized] [bh1101A@/dev/pts/4 connected] FreeBSD/amd64 (FreeBSD-11) (ttyu0) login: ======================== Even if the VM is not yet running you can connect to the server, and then start the VM, to all of its booting output. Or interact with the loader. It also works to actually install a VM from CD. Not that on 10.0 installs the default console is OFF, en needs to be edited in /etc/ttys to ttyu0 "/usr/libexec/getty std.9600" vt100 on secure or ttyu0 "/usr/libexec/getty std.9600" vt100 onifconsole secure For more recent relesase where init(1) understand onifconsole Probably there are some more things to do, like getting rtty and ssh to be silent on login, etc.... But these are the essentials. Rinse and repeat steps 2, 4 and 6 to create a new access. After which you net to restart the server to get it to learn a new device. And even though I'm a CLI junky .... Next step would to see if we can use all kinds of java(script) sshlibs to connect, so you can embed this in a management webconsole. Enjoy, --WjW From owner-freebsd-virtualization@FreeBSD.ORG Fri Jan 23 12:37:35 2015 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id CD975B96 for ; Fri, 23 Jan 2015 12:37:35 +0000 (UTC) Received: from ustc.edu.cn (email6.ustc.edu.cn [IPv6:2001:da8:d800::8]) by mx1.freebsd.org (Postfix) with ESMTP id E9EFBDCF for ; Fri, 23 Jan 2015 12:37:34 +0000 (UTC) Received: from freebsd.my.domain (unknown [58.211.218.74]) by newmailweb.ustc.edu.cn (Coremail) with SMTP id LkAmygA3FxSCQMJUYiyvAg--.21535S2; Fri, 23 Jan 2015 20:37:27 +0800 (CST) From: Tiwei Bie To: freebsd-virtualization@freebsd.org Subject: [PATCH] Add netmap support to bhyve Date: Sat, 24 Jan 2015 04:37:20 +0800 Message-Id: <1422045440-9410-1-git-send-email-btw@mail.ustc.edu.cn> X-Mailer: git-send-email 2.1.2 X-CM-TRANSID: LkAmygA3FxSCQMJUYiyvAg--.21535S2 X-Coremail-Antispam: 1UD129KBjvJXoW3ury7Ww18Kw1UGF4UWw1fXrb_yoWDZr4UpF W3A343tr40qayDWFZ3tFZ8Ar1aqrsYv3y3u3yrJ3s2kryUCr1xKFy2y39Yy34fCrZrXF1x Grn8AFy8tws8ZFDanT9S1TB71UUUUUUqnTZGkaVYY2UrUUUUjbIjqfuFe4nvWSU5nxnvy2 9KBjDU0xBIdaVrnRJUUUvlb7Iv0xC_Kw4lb4IE77IF4wAFF20E14v26r1j6r4UM7CY07I2 0VC2zVCF04k26cxKx2IYs7xG6rWj6s0DM7CIcVAFz4kK6r1j6r18M280x2IEY4vEnII2Ix kI6r1a6r45M28lY4IEw2IIxxk0rwA2F7IY1VAKz4vEj48ve4kI8wA2z4x0Y4vE2Ix0cI8I cVAFwI0_Ar0_tr1l84ACjcxK6xIIjxv20xvEc7CjxVAFwI0_Gr1j6F4UJwA2z4x0Y4vEx4 A2jsIE14v26rxl6s0DM28EF7xvwVC2z280aVCY1x0267AKxVW0oVCq3wAS0I0E0xvYzxvE 52x082IY62kv0487Mc02F40EFcxC0VAKzVAqx4xG6I80ewAv7VC0I7IYx2IY67AKxVWUXV WUAwAv7VC2z280aVAFwI0_Gr1j6F4UJwAm72CE4IkC6x0Yz7v_Jr0_Gr1lF7xvr2IYc2Ij 64vIr41lc2xSY4AK67AK6ryUMxAIw28IcxkI7VAKI48JMxC20s026xCaFVCjc4AY6r1j6r 4UMI8I3I0E5I8CrVAFwI0_Jr0_Jr4lx2IqxVCjr7xvwVAFwI0_JrI_JrWlx4CE17CEb7AF 67AKxVWUJVWUXwCIc40Y0x0EwIxGrwCI42IY6xIIjxv20xvE14v26r1j6r1xMIIF0xvE2I x0cI8IcVCY1x0267AKxVWUJVW8JwCI42IY6xAIw20EY4v20xvaj40_WFyUJVCq3wCI42IY 6I8E87Iv67AKxVWUJVW8JwCI42IY6I8E87Iv6xkF7I0E14v26r1j6r4UYxBIdaVFxhVjvj DU0xZFpf9x07baQ6JUUUUU= X-CM-SenderInfo: xewzqzxdloh3xvwfhvlgxou0/1tbiAQUSAVQhl+o6HwAGs8 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 23 Jan 2015 12:37:36 -0000 Hello everyone, I saw that there is an open task of adding Netmap support to bhyve in the latest status report of FreeBSD [1]. So I implement the netmap support for bhyve, and following is the patch. Hope this can be helpful! diff --git a/usr.sbin/bhyve/pci_virtio_net.c b/usr.sbin/bhyve/pci_virtio_net.c index 5ac9ecd..d7c8e84 100644 --- a/usr.sbin/bhyve/pci_virtio_net.c +++ b/usr.sbin/bhyve/pci_virtio_net.c @@ -35,6 +35,10 @@ __FBSDID("$FreeBSD$"); #include #include #include +#ifndef NETMAP_WITH_LIBS +#define NETMAP_WITH_LIBS +#endif +#include #include #include @@ -132,6 +136,8 @@ struct pci_vtnet_softc { struct mevent *vsc_mevp; int vsc_tapfd; + struct nm_desc *vsc_nmd; + int vsc_rx_ready; volatile int resetting; /* set and checked outside lock */ @@ -148,6 +154,10 @@ struct pci_vtnet_softc { pthread_mutex_t tx_mtx; pthread_cond_t tx_cond; int tx_in_progress; + + void (*pci_vtnet_rx)(struct pci_vtnet_softc *sc); + void (*pci_vtnet_tx)(struct pci_vtnet_softc *sc, struct iovec *iov, + int iovcnt, int len); }; static void pci_vtnet_reset(void *); @@ -369,14 +379,208 @@ pci_vtnet_tap_rx(struct pci_vtnet_softc *sc) vq_endchains(vq, 1); } +static int +pci_vtnet_netmap_writev(struct nm_desc *nmd, struct iovec *iov, int iovcnt) +{ + int r, i; + int len = 0; + + for (r = nmd->cur_tx_ring; ; ) { + struct netmap_ring *ring = NETMAP_TXRING(nmd->nifp, r); + uint32_t cur, idx; + char *buf; + + if (nm_ring_empty(ring)) { + r++; + if (r > nmd->last_tx_ring) + r = nmd->first_tx_ring; + if (r == nmd->cur_rx_ring) + break; + continue; + } + cur = ring->cur; + idx = ring->slot[cur].buf_idx; + buf = NETMAP_BUF(ring, idx); + + for (i = 0; i < iovcnt; i++) { + memcpy(&buf[len], iov[i].iov_base, iov[i].iov_len); + len += iov[i].iov_len; + } + ring->slot[cur].len = len; + ring->head = ring->cur = nm_ring_next(ring, cur); + nmd->cur_tx_ring = r; + ioctl(nmd->fd, NIOCTXSYNC, NULL); + break; + } + + return (len); +} + +static inline int +pci_vtnet_netmap_readv(struct nm_desc *nmd, struct iovec *iov, int iovcnt) +{ + int len = 0; + int i = 0; + int r; + + for (r = nmd->cur_rx_ring; ; ) { + struct netmap_ring *ring = NETMAP_RXRING(nmd->nifp, r); + uint32_t cur, idx; + char *buf; + size_t left; + + if (nm_ring_empty(ring)) { + r++; + if (r > nmd->last_rx_ring) + r = nmd->first_rx_ring; + if (r == nmd->cur_rx_ring) + break; + continue; + } + cur = ring->cur; + idx = ring->slot[cur].buf_idx; + buf = NETMAP_BUF(ring, idx); + left = ring->slot[cur].len; + + for (i = 0; i < iovcnt && left > 0; i++) { + if (iov[i].iov_len > left) + iov[i].iov_len = left; + memcpy(iov[i].iov_base, &buf[len], iov[i].iov_len); + len += iov[i].iov_len; + left -= iov[i].iov_len; + } + ring->head = ring->cur = nm_ring_next(ring, cur); + nmd->cur_rx_ring = r; + ioctl(nmd->fd, NIOCRXSYNC, NULL); + break; + } + for (; i < iovcnt; i++) + iov[i].iov_len = 0; + + return (len); +} + +/* + * Called to send a buffer chain out to the vale port + */ static void -pci_vtnet_tap_callback(int fd, enum ev_type type, void *param) +pci_vtnet_netmap_tx(struct pci_vtnet_softc *sc, struct iovec *iov, int iovcnt, + int len) +{ + static char pad[60]; /* all zero bytes */ + + if (sc->vsc_nmd == NULL) + return; + + /* + * If the length is < 60, pad out to that and add the + * extra zero'd segment to the iov. It is guaranteed that + * there is always an extra iov available by the caller. + */ + if (len < 60) { + iov[iovcnt].iov_base = pad; + iov[iovcnt].iov_len = 60 - len; + iovcnt++; + } + (void) pci_vtnet_netmap_writev(sc->vsc_nmd, iov, iovcnt); +} + +static void +pci_vtnet_netmap_rx(struct pci_vtnet_softc *sc) +{ + struct iovec iov[VTNET_MAXSEGS], *riov; + struct vqueue_info *vq; + void *vrx; + int len, n; + + /* + * Should never be called without a valid netmap descriptor + */ + assert(sc->vsc_nmd != NULL); + + /* + * But, will be called when the rx ring hasn't yet + * been set up or the guest is resetting the device. + */ + if (!sc->vsc_rx_ready || sc->resetting) { + /* + * Drop the packet and try later. + */ + (void) nm_nextpkt(sc->vsc_nmd, (void *)dummybuf); + return; + } + + /* + * Check for available rx buffers + */ + vq = &sc->vsc_queues[VTNET_RXQ]; + vq_startchains(vq); + if (!vq_has_descs(vq)) { + /* + * Drop the packet and try later. Interrupt on + * empty, if that's negotiated. + */ + (void) nm_nextpkt(sc->vsc_nmd, (void *)dummybuf); + vq_endchains(vq, 1); + return; + } + + do { + /* + * Get descriptor chain. + */ + n = vq_getchain(vq, iov, VTNET_MAXSEGS, NULL); + assert(n >= 1 && n <= VTNET_MAXSEGS); + + /* + * Get a pointer to the rx header, and use the + * data immediately following it for the packet buffer. + */ + vrx = iov[0].iov_base; + riov = rx_iov_trim(iov, &n, sc->rx_vhdrlen); + + len = pci_vtnet_netmap_readv(sc->vsc_nmd, riov, n); + + if (len == 0) { + /* + * No more packets, but still some avail ring + * entries. Interrupt if needed/appropriate. + */ + vq_endchains(vq, 0); + return; + } + + /* + * The only valid field in the rx packet header is the + * number of buffers if merged rx bufs were negotiated. + */ + memset(vrx, 0, sc->rx_vhdrlen); + + if (sc->rx_merge) { + struct virtio_net_rxhdr *vrxh; + + vrxh = vrx; + vrxh->vrh_bufs = 1; + } + + /* + * Release this chain and handle more chains. + */ + vq_relchain(vq, len + sc->rx_vhdrlen); + } while (vq_has_descs(vq)); + + /* Interrupt if needed, including for NOTIFY_ON_EMPTY. */ + vq_endchains(vq, 1); +} + +static void +pci_vtnet_rx_callback(int fd, enum ev_type type, void *param) { struct pci_vtnet_softc *sc = param; pthread_mutex_lock(&sc->rx_mtx); sc->rx_in_progress = 1; - pci_vtnet_tap_rx(sc); + sc->pci_vtnet_rx(sc); sc->rx_in_progress = 0; pthread_mutex_unlock(&sc->rx_mtx); @@ -417,7 +621,7 @@ pci_vtnet_proctx(struct pci_vtnet_softc *sc, struct vqueue_info *vq) } DPRINTF(("virtio: packet send, %d bytes, %d segs\n\r", plen, n)); - pci_vtnet_tap_tx(sc, &iov[1], n - 1, plen); + sc->pci_vtnet_tx(sc, &iov[1], n - 1, plen); /* chain is processed, release it and set tlen */ vq_relchain(vq, tlen); @@ -530,6 +734,67 @@ pci_vtnet_parsemac(char *mac_str, uint8_t *mac_addr) return (0); } +static void +pci_vtnet_tap_setup(struct pci_vtnet_softc *sc, char *devname) +{ + char tbuf[80]; + + strcpy(tbuf, "/dev/"); + strlcat(tbuf, devname, sizeof(tbuf)); + + sc->pci_vtnet_rx = pci_vtnet_tap_rx; + sc->pci_vtnet_tx = pci_vtnet_tap_tx; + + sc->vsc_tapfd = open(tbuf, O_RDWR); + if (sc->vsc_tapfd == -1) { + WPRINTF(("open of tap device %s failed\n", tbuf)); + return; + } + + /* + * Set non-blocking and register for read + * notifications with the event loop + */ + int opt = 1; + if (ioctl(sc->vsc_tapfd, FIONBIO, &opt) < 0) { + WPRINTF(("tap device O_NONBLOCK failed\n")); + close(sc->vsc_tapfd); + sc->vsc_tapfd = -1; + } + + sc->vsc_mevp = mevent_add(sc->vsc_tapfd, + EVF_READ, + pci_vtnet_rx_callback, + sc); + if (sc->vsc_mevp == NULL) { + WPRINTF(("Could not register event\n")); + close(sc->vsc_tapfd); + sc->vsc_tapfd = -1; + } +} + +static void +pci_vtnet_netmap_setup(struct pci_vtnet_softc *sc, char *ifname) +{ + sc->pci_vtnet_rx = pci_vtnet_netmap_rx; + sc->pci_vtnet_tx = pci_vtnet_netmap_tx; + + sc->vsc_nmd = nm_open(ifname, NULL, 0, 0); + if (sc->vsc_nmd == NULL) { + WPRINTF(("open of netmap device %s failed\n", ifname)); + return; + } + + sc->vsc_mevp = mevent_add(sc->vsc_nmd->fd, + EVF_READ, + pci_vtnet_rx_callback, + sc); + if (sc->vsc_mevp == NULL) { + WPRINTF(("Could not register event\n")); + nm_close(sc->vsc_nmd); + sc->vsc_nmd = NULL; + } +} static int pci_vtnet_init(struct vmctx *ctx, struct pci_devinst *pi, char *opts) @@ -565,8 +830,8 @@ pci_vtnet_init(struct vmctx *ctx, struct pci_devinst *pi, char *opts) */ mac_provided = 0; sc->vsc_tapfd = -1; + sc->vsc_nmd = NULL; if (opts != NULL) { - char tbuf[80]; int err; devname = vtopts = strdup(opts); @@ -581,36 +846,12 @@ pci_vtnet_init(struct vmctx *ctx, struct pci_devinst *pi, char *opts) mac_provided = 1; } - strcpy(tbuf, "/dev/"); - strlcat(tbuf, devname, sizeof(tbuf)); + if (strncmp(devname, "vale", 4) == 0) + pci_vtnet_netmap_setup(sc, devname); + if (strncmp(devname, "tap", 3) == 0) + pci_vtnet_tap_setup(sc, devname); free(devname); - - sc->vsc_tapfd = open(tbuf, O_RDWR); - if (sc->vsc_tapfd == -1) { - WPRINTF(("open of tap device %s failed\n", tbuf)); - } else { - /* - * Set non-blocking and register for read - * notifications with the event loop - */ - int opt = 1; - if (ioctl(sc->vsc_tapfd, FIONBIO, &opt) < 0) { - WPRINTF(("tap device O_NONBLOCK failed\n")); - close(sc->vsc_tapfd); - sc->vsc_tapfd = -1; - } - - sc->vsc_mevp = mevent_add(sc->vsc_tapfd, - EVF_READ, - pci_vtnet_tap_callback, - sc); - if (sc->vsc_mevp == NULL) { - WPRINTF(("Could not register event\n")); - close(sc->vsc_tapfd); - sc->vsc_tapfd = -1; - } - } } /* -- 1.9.3 (Apple Git-50) ---------------------------------------------------------------- Following are the simple instructions to test the patch by creating two virtual machines: #1. Launch the first vm with a NIC configured as a vale port: $ bhyve -A -H -P \ -s 0:0,hostbridge \ -s 1:0,lpc \ -s 2,virtio-blk,./disk-vm1 \ -s 3:0,virtio-net,vale0:vm1 \ -l com1,stdio \ -m 1G -c 2 vm1 #2. Launch the second vm with a NIC configured as a vale port: $ bhyve -A -H -P \ -s 0:0,hostbridge \ -s 1:0,lpc \ -s 2,virtio-blk,./disk-vm2 \ -s 3:0,virtio-net,vale0:vm2 \ -l com1,stdio \ -m 1G -c 2 vm2 Both of vale0:vm1 and vale0:vm2 are the ports of vale0 switch. #3. Setup the network in each vm: $ ifconfig vtnet0 192.168.1.1 up # in the 1st vm $ ifconfig vtnet0 192.168.1.2 up # in the 2nd vm #4. Ping test # ping -c 5 192.168.1.1 # in the 2nd vm PING 192.168.1.1 (192.168.1.1): 56 data bytes 64 bytes from 192.168.1.1: icmp_seq=0 ttl=64 time=0.513 ms 64 bytes from 192.168.1.1: icmp_seq=1 ttl=64 time=0.356 ms 64 bytes from 192.168.1.1: icmp_seq=2 ttl=64 time=0.523 ms 64 bytes from 192.168.1.1: icmp_seq=3 ttl=64 time=0.362 ms 64 bytes from 192.168.1.1: icmp_seq=4 ttl=64 time=0.532 ms --- 192.168.1.1 ping statistics --- 5 packets transmitted, 5 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 0.356/0.457/0.532/0.080 ms [1] https://www.freebsd.org/news/status/report-2014-10-2014-12.html#bhyve Tiwei Bie