From nobody Mon Jul 26 16:58:17 2021 X-Original-To: jail@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id C021312AFFFB for ; Mon, 26 Jul 2021 16:58:17 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4GYR2K3hZpz4YHJ for ; Mon, 26 Jul 2021 16:58:17 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 5CEC01AB19 for ; Mon, 26 Jul 2021 16:58:17 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 16QGwH6T045108 for ; Mon, 26 Jul 2021 16:58:17 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 16QGwH2n045107 for jail@FreeBSD.org; Mon, 26 Jul 2021 16:58:17 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: jail@FreeBSD.org Subject: [Bug 251046] bhyve PCI passthrough does not work inside jail Date: Mon, 26 Jul 2021 16:58:17 +0000 X-Bugzilla-Reason: CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 12.2-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: markj@FreeBSD.org X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussion about FreeBSD jail(8) List-Archive: https://lists.freebsd.org/archives/freebsd-jail List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-jail@freebsd.org MIME-Version: 1.0 X-ThisMailContainsUnwantedMimeParts: N https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D251046 --- Comment #8 from Mark Johnston --- Sorry for the delayed follow up. I wrote some patches to remove the need f= or /dev/io: https://reviews.freebsd.org/D31307 https://reviews.freebsd.org/D31308 Testing would be appreciated. This does not remove the dependency on /dev/= mem yet. I am very skeptical that jailing bhyve with PCI passthrough enabled provides any meaningful security. /dev/pci allows a jailed root to access all PCI(e) devices in the system. Jails can be a useful deployment mechanism though, = so I think we should better support their integration with bhyve. --=20 You are receiving this mail because: You are on the CC list for the bug.= From nobody Mon Jul 26 18:34:00 2021 X-Original-To: jail@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 36FF512A8F00 for ; Mon, 26 Jul 2021 18:34:01 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4GYT8n0WNcz3GCL for ; Mon, 26 Jul 2021 18:34:01 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id E90531BDEE for ; Mon, 26 Jul 2021 18:34:00 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 16QIY0u7097977 for ; Mon, 26 Jul 2021 18:34:00 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 16QIY07x097976 for jail@FreeBSD.org; Mon, 26 Jul 2021 18:34:00 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: jail@FreeBSD.org Subject: [Bug 251046] bhyve PCI passthrough does not work inside jail Date: Mon, 26 Jul 2021 18:34:00 +0000 X-Bugzilla-Reason: CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 12.2-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: bz@FreeBSD.org X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussion about FreeBSD jail(8) List-Archive: https://lists.freebsd.org/archives/freebsd-jail List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-jail@freebsd.org MIME-Version: 1.0 X-ThisMailContainsUnwantedMimeParts: N https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D251046 Bjoern A. Zeeb changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |bz@FreeBSD.org --- Comment #9 from Bjoern A. Zeeb --- The /dev/mem ones could probably be PCIOCBARMMAP if that could be/is locked down enough? But I assume all the checks needed are in place (now) somewhe= re for the IO ioctl? --=20 You are receiving this mail because: You are on the CC list for the bug.= From nobody Mon Jul 26 19:15:20 2021 X-Original-To: freebsd-jail@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 6F00912B7630 for ; Mon, 26 Jul 2021 19:15:33 +0000 (UTC) (envelope-from jacques+freebsd@foucry.net) Received: from mail.foucry.net (mail.foucry.net [IPv6:2a01:4f9:4a:1fd8::17]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4GYV4h25qnz3hM5 for ; Mon, 26 Jul 2021 19:15:32 +0000 (UTC) (envelope-from jacques+freebsd@foucry.net) Received: from mail.foucry.net (unknown [192.168.12.17]) by mail.foucry.net (Postfix) with ESMTP id E87A610931 for ; Mon, 26 Jul 2021 19:15:22 +0000 (UTC) X-Virus-Scanned: amavisd-new at foucry.net Received: from mail.foucry.net ([192.168.12.17]) by mail.foucry.net (mail.foucry.net [192.168.12.17]) (amavisd-new, port 10024) with ESMTP id nmfu0FF-hhR9 for ; Mon, 26 Jul 2021 19:15:22 +0000 (UTC) Received: by mail.foucry.net (Postfix, from userid 58) id 6665B107EC; Mon, 26 Jul 2021 19:15:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=foucry.net; s=dkim; t=1627326922; bh=KYux0zox+WgkUWLTLGZFx9Pfuj6kopwo+XdzX+gPhAg=; h=Date:From:To:Cc:Subject:References:In-Reply-To; b=Q/54UEFUQgOZ1phc/DrJ50GM6gh7ILpzXhIl8yK611r8b9s8wbv7pbpoQf4P1W+KR pjXXVAWKQZF8zpx+g9ofcoS0/Jemj+D2RMQPi0v/F0QGk8s356tF+PjuOq6/+Mocxb 907FB0N579sWlh7Ncuy3WDx297QZed7Jy0WZvKTU= Received: from mithril.foucry.net (82-65-174-130.subs.proxad.net [82.65.174.130]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mail.foucry.net (Postfix) with ESMTPSA id 209A5107EB; Mon, 26 Jul 2021 19:15:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=foucry.net; s=dkim; t=1627326922; bh=KYux0zox+WgkUWLTLGZFx9Pfuj6kopwo+XdzX+gPhAg=; h=Date:From:To:Cc:Subject:References:In-Reply-To; b=Q/54UEFUQgOZ1phc/DrJ50GM6gh7ILpzXhIl8yK611r8b9s8wbv7pbpoQf4P1W+KR pjXXVAWKQZF8zpx+g9ofcoS0/Jemj+D2RMQPi0v/F0QGk8s356tF+PjuOq6/+Mocxb 907FB0N579sWlh7Ncuy3WDx297QZed7Jy0WZvKTU= Received: from mithril.foucry.net (localhost [IPv6:::1]) by mithril.foucry.net (Postfix) with ESMTPS id 347D41361; Mon, 26 Jul 2021 21:15:21 +0200 (CEST) Date: Mon, 26 Jul 2021 21:15:20 +0200 From: Jacques Foucry To: infoomatic Cc: freebsd-jail@freebsd.org Subject: Re: iocage, vnet jail does not go outside Message-ID: Mail-Followup-To: infoomatic , freebsd-jail@freebsd.org References: <40b7782d-9d5c-099a-ed58-4476b3523d7a@gmx.at> <3c0bcf3e-541f-5add-47cd-9457d4e5dc85@gmx.at> List-Id: Discussion about FreeBSD jail(8) List-Archive: https://lists.freebsd.org/archives/freebsd-jail List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-jail@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <3c0bcf3e-541f-5add-47cd-9457d4e5dc85@gmx.at> X-Operating-System: FreeBSD X-Rspamd-Queue-Id: 4GYV4h25qnz3hM5 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=fail ("headers rsa verify failed") header.d=foucry.net header.s=dkim header.b="Q/54UEFU"; dkim=fail ("headers rsa verify failed") header.d=foucry.net header.s=dkim header.b="Q/54UEFU"; dmarc=pass (policy=none) header.from=foucry.net; spf=pass (mx1.freebsd.org: domain of jacques@foucry.net designates 2a01:4f9:4a:1fd8::17 as permitted sender) smtp.mailfrom=jacques@foucry.net X-Spamd-Result: default: False [-2.80 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; ARC_NA(0.00)[]; RCVD_COUNT_FIVE(0.00)[6]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+mx]; R_DKIM_REJECT(0.00)[foucry.net:s=dkim]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-jail@freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MID_RHS_MATCH_FROMTLD(0.00)[]; DKIM_TRACE(0.00)[foucry.net:-]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(0.00)[foucry.net,none]; NEURAL_HAM_SHORT(-1.00)[-1.000]; DMARC_POLICY_ALLOW_WITH_FAILURES(-0.50)[]; FREEMAIL_TO(0.00)[gmx.at]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:24940, ipnet:2a01:4f9::/32, country:DE]; TAGGED_FROM(0.00)[freebsd]; MAILMAN_DEST(0.00)[freebsd-jail]; RECEIVED_SPAMHAUS_PBL(0.00)[82.65.174.130:received] X-ThisMailContainsUnwantedMimeParts: N Le samedi 24 juil. 2021 à 23:48:26 (+0200), infoomatic à écrit: Hello, > Hi, > > sorry to hear that. That's life :-) and I learn a lot form my mistake and your help. > > I use the tools from the FreeBSD base system, they work great, and I > encourage all newbies to use the tools from the base systems - and > recommend reading the parts of the handbook and the man pages of jail > and jail.conf I use to. I alose read Michael W. Lucas Mastery Jails bookṡ > > Here are the relevant parts of my config: > > rc.conf: > > cloned_interfaces="bridge0" > > ifconfig_bridge0="inet 192.168.1.1 netmask 255.255.255.0 up" > > pf.conf: > > nat pass on em0 proto tcp from {192.168.1.201} to any -> pu.bl.ic.ip > > and the jail.conf: > > example { >     host.hostname = example; >     vnet; >     vnet.interface = "epair201b"; >     path ="/jails/$name"; >     exec.prestart += "ifconfig epair201 create"; >     exec.prestart += "ifconfig epair201a up"; >     exec.prestart += "ifconfig bridge0 addm epair201a"; >     exec.prestop += "ifconfig epair201b -vnet $name"; >     exec.poststop += "ifconfig epair201a destroy"; > > } > > and the /jails/example/etc/rc.conf: > > ifconfig_epair201b="inet 192.168.1.201 netmask 255.255.255.0" > defaultrouter="192.168.1.1" > hope this helps, Of course it helps. And as I understood about إaving or not em0 into the bridge. Without your are sure that your jail CANNOT communicate with the external world (useful for a database jail for example), and with your jail CAN communicate with the external world (useful for a weⅺsie ou mail jail). In my case, I would like to have a VNET jail that can dialog with the World. So, from your sample I add em0 the bridge and give it an IPv4 address, but it did not work. In any case, thanks for your help and the time you spent on my stupid problem. Btw I read all the other answer and try to make a mixupo on my brain with all this informations. Thanks to all. > > > >> iocage autoatically creates a bridge with your physical interface and > >> the vnet interface. Imho this is wrong behaviour so I quit using iocage, > >> however, there is a workaround, for more info see [1] > > > > I read carfully the issue your pointed and it appears that the > > vnet_default_interface parameter set to auto, em0 is added to the bridge, set > > to none, em0 is not added to the bridge. > > > > So I stopped my jail, destroy bridge0 interface, set vnet_default_interface to > > none and restart the jail. > > > > As exepected em0 is not in the bridge any more: > > > > bridge0: flags=8843 metric 0 mtu 1500 > > description: jails-bridge > > ether 58:9c:fc:10:ed:66 > > inet 10.0.10.1 netmask 0xffffff00 broadcast 10.0.10.255 > > id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 > > maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 > > root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 > > member: vnet0.657 flags=143 > > ifmaxaddr 0 port 6 priority 128 path cost 2000 > > groups: bridge > > nd6 options=9 > > > > Since from the jail I cannot ping anything, from outside I cannot connect to > > the jail and from the jail I cannot connect to outside host. > > > > In fact, see quickly, the situation is worst. > > > > I did not look at the routing tables yet (too many other things to do). > > > > As I understood your did not use iocage any more. Did you use the "raw" > > method (ie /etc/jail.conf)? If yes, I am really interested of "picture" of > > your configurætion. > > > > To be honest, I used to try the "raw" method whithout success before tring > > iocage. > > > > Thanks for your time and advices. > -- Jacques Foucry From nobody Tue Jul 27 13:44:42 2021 X-Original-To: jail@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id A826812BD7EE for ; Tue, 27 Jul 2021 13:44:42 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4GYyhT6G5vz4W5b for ; Tue, 27 Jul 2021 13:44:41 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id BFCFF37D6 for ; Tue, 27 Jul 2021 13:44:41 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 16RDifwG095162 for ; Tue, 27 Jul 2021 13:44:41 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 16RDifRQ095161 for jail@FreeBSD.org; Tue, 27 Jul 2021 13:44:41 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: jail@FreeBSD.org Subject: [Bug 251046] bhyve PCI passthrough does not work inside jail Date: Tue, 27 Jul 2021 13:44:42 +0000 X-Bugzilla-Reason: CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 12.2-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: markj@FreeBSD.org X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussion about FreeBSD jail(8) List-Archive: https://lists.freebsd.org/archives/freebsd-jail List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-jail@freebsd.org MIME-Version: 1.0 X-ThisMailContainsUnwantedMimeParts: N https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D251046 --- Comment #10 from Mark Johnston --- (In reply to Bjoern A. Zeeb from comment #9) > The /dev/mem ones could probably be PCIOCBARMMAP if that could be/is lock= ed down enough? Yes, it seems possible. > But I assume all the checks needed are in place (now) somewhere for the I= O ioctl? I'm not sure if I understand the question. The new ioctl limits accesses to the specified BAR and verifies that the accesses is within bounds. The /de= v/io interface permits access to any system I/O port. --=20 You are receiving this mail because: You are on the CC list for the bug.=