Date: Mon, 3 Jun 2013 16:31:26 +0000 From: "Teske, Devin" <Devin.Teske@fisglobal.com> To: Pietro Paolini <pulsarpietro@aol.com> Cc: Devin Teske <dteske@freebsd.org>, FreeBSD Questions <freebsd-questions@freebsd.org> Subject: Re: VIMAGE Message-ID: <13CA24D6AB415D428143D44749F57D7201F6EBFF@ltcfiswmsgmb21> In-Reply-To: <93DE0F1F-FF8C-4A48-BE9B-70C0F0B84AE8@aol.com> References: <DB90C1DC-66E4-4429-A888-44F4F9E4B98B@aol.com> <13CA24D6AB415D428143D44749F57D7201F68CBD@ltcfiswmsgmb21> <DA96E7A7-C419-4C73-A27B-D02BAB2CBE4E@aol.com> <13CA24D6AB415D428143D44749F57D7201F6B5F0@ltcfiswmsgmb21> <13CA24D6AB415D428143D44749F57D7201F6BCEB@ltcfiswmsgmb21> <93DE0F1F-FF8C-4A48-BE9B-70C0F0B84AE8@aol.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Jun 3, 2013, at 6:33 AM, Pietro Paolini wrote: >=20 > On Jun 1, 2013, at 5:26 AM, "Teske, Devin" <Devin.Teske@fisglobal.com> wr= ote: >=20 >>=20 >> On May 31, 2013, at 3:05 PM, Teske, Devin wrote: >>=20 >>>=20 >>> On May 31, 2013, at 1:48 AM, Pietro Paolini wrote: >>>=20 >>>>=20 >>>> On May 30, 2013, at 6:25 PM, "Teske, Devin" <Devin.Teske@fisglobal.com= > wrote: >>>>=20 >>>>>=20 >>>>> On May 30, 2013, at 3:35 AM, Pietro Paolini wrote: >>>>>=20 >>>>>> Hello all, >>>>>>=20 >>>>>> I am a new bye on the FreeBSD and I am looking at the VIMAGE feature= s experiencing some problems. >>>>>> I added the options : >>>>>> VIMAGE >>>>>> if_bridge >>>>>>=20 >>>>>> and I removed >>>>>> STCP >>>>>>=20 >>>>>> then I recompiled my kernel and install it. >>>>>>=20 >>>>>> After that, following this tutorial https://urldefense.proofpoint.co= m/v1/url?u=3Dhttp://imunes.tel.fer.hr/virtnet/eurobsdcon07_tutorial.pdf&k= =3D%2FbkpAUdJWZuiTILCq%2FFnQg%3D%3D%0A&r=3DMrjs6vR4%2Faj2Ns9%2FssHJjg%3D%3D= %0A&m=3Dxe0XNgnKBiT9v8HzxwWwnNMOVN3YdEHmTsIZfFoQA9Y%3D%0A&s=3Db271820faf310= ac274ded8c3135b4931f2a5278e78ec1a66ce6a5ab971ff64f5 I tried the "Exercise 2= " which consist on=20 >>>>>> the following commands: >>>>>>=20 >>>>>> vimage -c n1 >>>>>> vimage -c n2 >>>>>> ngctl mkpeer efface ether ether >>>>>> ngctl mkpeer efface ether ether >>>>>=20 >>>>> Don't you just love autocorrect? (does the same thing to me=85 turns = "eiface" into "efface") >>>>>=20 >>>>>=20 >>>>>> ngctl mkpeer em0: bridge lower link0 >>>>>=20 >>>>> Looks good. >>>>>=20 >>>>>=20 >>>>>> ngctl name em0:lower bridge0 >>>>>=20 >>>>> I usually do my "connect" before the "name"=85 but shouldn't matter. = Should work all the same. >>>>>=20 >>>>>=20 >>>>>> ngctl connect em0: bridge0: upper link1 >>>>>=20 >>>>> This looks wrong to me. >>>>>=20 >>>>> I'd expect: >>>>>=20 >>>>> ngctl connect em0: bridge0:lower upper link1 >>>>>=20 >>>>=20 >>>>=20 >>>> Many thanks for the answer Devin, >>>> when I try to use that last command I receive: >>>>=20 >>>> ngctl connect em0: bridge0:lower upper link1 >>>> ngctl: send msg: Invalid argument >>>>=20 >>>> What's wrong ? >>>>=20 >>>=20 >>> Let's start from scratch on a freshly booted box=85 >>>=20 >>> dteske@scu0a.jbsd.vicor.com ~ $ sudo ngctl ls -l >>> [sudo] Password: >>> There are 4 total nodes: >>> Name: em0 Type: ether ID: 00000002 Num hooks: 0 >>> Name: em1 Type: ether ID: 00000003 Num hooks: 0 >>> Name: ngctl1719 Type: socket ID: 00000004 Num hooks: 0 >>> Name: msk0 Type: ether ID: 00000001 Num hooks: 0 >>>=20 >>> Ok=85 we have an "ether" type node for each of our physical adapters (t= hese are provided by ng_ether(4); you didn't have to do anything to get the= se nodes). >>>=20 >>> We also have a single "socket" type node. This is the "ngctl" connectio= n to the netgraph subsystem (you can learn more by reading ng_socket(4)). >>>=20 >>> Here's the corresponding hardware behind em0, em1, and msk0: >>>=20 >>> =3D=3D=3D >>>=20 >>> dteske@scu0a.jbsd.vicor.com ~ $ grep '\(em\|e1000phy\|mskc\?\)[[:digit:= ]]' /var/run/dmesg.boot >>> mskc0: <Marvell Yukon 88E8050 Gigabit Ethernet> port 0xdc00-0xdcff mem = 0xfcffc000-0xfcffffff irq 16 at device 0.0 on pci5 >>> msk0: <Marvell Technology Group Ltd. Yukon EC Id 0xb6 Rev 0x02> on mskc0 >>> msk0: Ethernet address: xx:xx:xx:xx:xx:xx >>> miibus0: <MII bus> on msk0 >>> e1000phy0: <Marvell 88E1111 Gigabit PHY> PHY 0 on miibus0 >>> e1000phy0: none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000b= aseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto >>> mskc0: [ITHREAD] >>> em0: <Intel(R) PRO/1000 Legacy Network Connection 1.0.3> port 0xec80-0x= ecbf mem 0xfebe0000-0xfebfffff irq 16 at device 4.0 on pci7 >>> em0: [FILTER] >>> em0: Ethernet address: xx:xx:xx:xx:xx:xx >>> em1: <Intel(R) PRO/1000 Legacy Network Connection 1.0.3> port 0xec00-0x= ec3f mem 0xfeba0000-0xfebbffff,0xfeb80000-0xfeb9ffff irq 18 at device 6.0 o= n pci7 >>> em1: [FILTER] >>> em1: Ethernet address: xx:xx:xx:xx:xx:xx >>> em0: link state changed to UP >>>=20 >>> =3D=3D=3D >>>=20 >>> Next, let's make a bridge (think of it as a big software switch that we= 're going to hook a bunch of interfaces; created, physical, or otherwise). >>>=20 >>> Since I'm doing this over an SSH connection (a mistake I made earlier t= oday), I'm not going to touch em0 (the adapter my SSH connection is using).= Creating the bridge on an actively configured PHY will knock it off the ne= t. This is not to say you can't have an active configuration on a bridged i= nterface=85 just that the creation of the bridge (something you should only= do once each time you boot) will disrupt an active connection. >>>=20 >>> So=85 >>>=20 >>> dteske@scu0a.jbsd.vicor.com ~ $ sudo ngctl mkpeer em1: bridge lower lin= k0 >>>=20 >>> NOTE: No output =3D=3D Success. >>>=20 >>> =3D=3D=3D >>>=20 >>> Now let's look at our handiwork=85 >>>=20 >>> dteske@scu0a.jbsd.vicor.com ~ $ sudo ngctl info em1:lower >>> Name: <unnamed> Type: bridge ID: 00000007 Num hooks: 1 >>> Local hook Peer name Peer type Peer ID Peer hook= =20=20=20=20=20=20 >>> ---------- --------- --------- ------- ---------= =20=20=20=20=20=20 >>> link0 em1 ether 00000003 lower=20= =20=20=20=20=20=20=20=20=20 >>>=20 >>>=20 >>> Ok, we see that the lower peer hook of the em1 ether-node goes off to s= omething named "link0". >>>=20 >>> To see where link0 is off-to=85 we need a full listing (back to "ngctl = ls -l"). >>>=20 >>>=20 >>> dteske@scu0a.jbsd.vicor.com ~ $ sudo ngctl ls -l >>> There are 5 total nodes: >>> Name: <unnamed> Type: bridge ID: 00000007 Num hooks: 1 >>> Local hook Peer name Peer type Peer ID Peer hook= =20=20=20=20=20=20 >>> ---------- --------- --------- ------- ---------= =20=20=20=20=20=20 >>> link0 em1 ether 00000003 lower=20= =20=20=20=20=20=20=20=20=20 >>> Name: em0 Type: ether ID: 00000002 Num hooks: 0 >>> Name: em1 Type: ether ID: 00000003 Num hooks: 1 >>> Local hook Peer name Peer type Peer ID Peer hook= =20=20=20=20=20=20 >>> ---------- --------- --------- ------- ---------= =20=20=20=20=20=20 >>> lower <unnamed> bridge 00000007 link0=20= =20=20=20=20=20=20=20=20=20 >>> Name: ngctl1762 Type: socket ID: 0000000b Num hooks: 0 >>> Name: msk0 Type: ether ID: 00000001 Num hooks: 0 >>>=20 >>>=20 >>> Matching "link0" in the first column to "link0" in the last-column, we = can see that this lower-link0 is to a bridge (with no name). >>>=20 >>> NOTE: When you're digesting the above output=85 it helps to imagine whi= tespace in between the nodes with their respective hooks and other nodes. F= uture pastes below will introduce such whitespace to make it easier to read. >>>=20 >>> =3D=3D=3D >>>=20 >>> Right now, the only way to refer to the bridge is by way of "em1:lower"= (because we created the bridge right on the lower hook of the em1 ether-no= de). >>>=20 >>> At this point, let's talk about naming. Giving our bridge a name is ent= irely optional, but greatly clarifies the output of both "ngctl ls -l" and = "ngctl dot". >>>=20 >>> dteske@scu0a.jbsd.vicor.com ~ $ sudo ngctl name em1:lower em1bridge >>> dteske@scu0a.jbsd.vicor.com ~ $ sudo ngctl ls -l >>> There are 5 total nodes: >>> Name: em0 Type: ether ID: 00000002 Num hooks: 0 >>>=20 >>> Name: em1 Type: ether ID: 00000003 Num hooks: 1 >>> Local hook Peer name Peer type Peer ID Peer hook= =20=20=20=20=20=20 >>> ---------- --------- --------- ------- ---------= =20=20=20=20=20=20 >>> lower em1bridge bridge 00000007 link0=20= =20=20=20=20=20=20=20=20=20 >>>=20 >>> Name: ngctl1831 Type: socket ID: 0000001a Num hooks: 0 >>>=20 >>> Name: em1bridge Type: bridge ID: 00000007 Num hooks: 1 >>> Local hook Peer name Peer type Peer ID Peer hook= =20=20=20=20=20=20 >>> ---------- --------- --------- ------- ---------= =20=20=20=20=20=20 >>> link0 em1 ether 00000003 lower=20= =20=20=20=20=20=20=20=20=20 >>>=20 >>> Name: msk0 Type: ether ID: 00000001 Num hooks: 0 >>>=20 >>> The new "em1bridge" name acts as an alias to "em1:lower" in future ngct= l commands. For example, "ngctl info em1:lower" and "ngctl info em1bridge" = can now be used interchangeably and produce the same results. >>>=20 >>> dteske@scu0a.jbsd.vicor.com ~ $ sudo ngctl info em1bridge: >>> Name: em1bridge Type: bridge ID: 00000007 Num hooks: 1 >>> Local hook Peer name Peer type Peer ID Peer hook= =20=20=20=20=20=20 >>> ---------- --------- --------- ------- ---------= =20=20=20=20=20=20 >>> link0 em1 ether 00000003 lower=20= =20=20=20=20=20=20=20=20=20 >>> dteske@scu0a.jbsd.vicor.com ~ $ sudo ngctl info em1:lower=20 >>> Name: em1bridge Type: bridge ID: 00000007 Num hooks: 1 >>> Local hook Peer name Peer type Peer ID Peer hook= =20=20=20=20=20=20 >>> ---------- --------- --------- ------- ---------= =20=20=20=20=20=20 >>> link0 em1 ether 00000003 lower=20= =20=20=20=20=20=20=20=20=20 >>>=20 >>> =3D=3D=3D >>>=20 >>> We're not done with the bridge yet. Because we foresee the possibility = that it might be nice to be able to communicate with the jail that we're go= ing to later hook into this bridge=85 we should hook the physical adapter's= "upper" hook into the bridge. >>>=20 >>> If you don't do this, you won't be able to (for example) ping a jail fr= om the host where the host has only the PHY and the jail has only a (yet un= created) eiface. Regardless of the fact that the bridge uses the PHY and th= e jail uses the bridge, to communicate with an IP that is configured on the= base host, you must hook the upper. >>>=20 >>> dteske@scu0a.jbsd.vicor.com ~ $ sudo ngctl connect em1: em1:lower upper= link1 >>>=20 >>> If you want to use the alias I set up earlier (of "em1bridge") that wor= ks too (just don't forget the colon at the end of the alias): >>>=20 >>> dteske@scu0a.jbsd.vicor.com ~ $ sudo ngctl connect em1: em1bridge: uppe= r link1 >>>=20 >>> Here's the results: >>>=20 >>> dteske@scu0a.jbsd.vicor.com ~ $ sudo ngctl ls -l >>> There are 5 total nodes: >>> Name: em0 Type: ether ID: 00000002 Num hooks: 0 >>>=20 >>> Name: em1 Type: ether ID: 00000003 Num hooks: 2 >>> Local hook Peer name Peer type Peer ID Peer hook= =20=20=20=20=20=20 >>> ---------- --------- --------- ------- ---------= =20=20=20=20=20=20 >>> upper em1bridge bridge 0000002a link1=20= =20=20=20=20=20=20=20=20=20 >>> lower em1bridge bridge 0000002a link0=20= =20=20=20=20=20=20=20=20=20 >>>=20 >>> Name: ngctl1874 Type: socket ID: 00000030 Num hooks: 0 >>>=20 >>> Name: em1bridge Type: bridge ID: 0000002a Num hooks: 2 >>> Local hook Peer name Peer type Peer ID Peer hook= =20=20=20=20=20=20 >>> ---------- --------- --------- ------- ---------= =20=20=20=20=20=20 >>> link1 em1 ether 00000003 upper=20= =20=20=20=20=20=20=20=20=20 >>> link0 em1 ether 00000003 lower=20= =20=20=20=20=20=20=20=20=20 >>>=20 >>> Name: msk0 Type: ether ID: 00000001 Num hooks: 0 >>>=20 >>>=20 >>> NOTE: Some of the Peer ID's have changed, because I wanted to test that= the alias could be used; I used "sudo ngctl shutdown em1bridge:" and re-ex= ecuted up to the point where I connect the em1:upper into the bridge=85 exc= ept this time using the alias of "em1bridge" instead of "em1:lower" (indeed= , you can use them interchangeably). >>>=20 >>> =3D=3D=3D >>>=20 >>> Ok=85 We've now done the hard part=85 which was to create and configure= a bridge that is usable by any new nodes we connect to it and also (if you= hooked the upper portion of em1 back into its own lower which is acting as= the bridge) the base machine can communicate with any of the forth-coming = jails (if on the same subnet at least). >>>=20 >>> There's an easy step that shouldn't be skipped though=85 >>>=20 >>> Before you can truly use this bridge with any other interfaces=85 >>>=20 >>> dteske@scu0a.jbsd.vicor.com ~ $ sudo ifconfig em1 up >>> dteske@scu0a.jbsd.vicor.com ~ $ sudo ngctl msg em1: setpromisc 1 >>> dteske@scu0a.jbsd.vicor.com ~ $ sudo ngctl msg em1: setautosrc 0 >>>=20 >>> A bridge cannot send packets out if the interface is down. >>> A bridge cannot work properly without promiscuous mode. >>> A bridge cannot send out packets for different addresses unless you tur= n off "setautosrc" >>>=20 >>> =3D=3D=3D >>>=20 >>> Let's create our first virtual NIC and connect it to the bridge. >>>=20 >>> dteske@scu0a.jbsd.vicor.com ~ $ sudo ngctl mkpeer em1bridge: eiface lin= k2 ether >>>=20 >>> This command did two things. It created a new "eiface" node (see ng_eif= ace(4)), and connected it to the bridge. >>>=20 >>> Let's have a look: >>>=20 >>> dteske@scu0a.jbsd.vicor.com ~ $ sudo ngctl ls -l >>> There are 6 total nodes: >>> Name: em0 Type: ether ID: 00000002 Num hooks: 0 >>>=20 >>> Name: em1 Type: ether ID: 00000003 Num hooks: 2 >>> Local hook Peer name Peer type Peer ID Peer hook= =20=20=20=20=20=20 >>> ---------- --------- --------- ------- ---------= =20=20=20=20=20=20 >>> upper em1bridge bridge 0000002a link1=20= =20=20=20=20=20=20=20=20=20 >>> lower em1bridge bridge 0000002a link0=20= =20=20=20=20=20=20=20=20=20 >>>=20 >>> Name: ngeth0 Type: eiface ID: 00000035 Num hooks: 1 >>> Local hook Peer name Peer type Peer ID Peer hook= =20=20=20=20=20=20 >>> ---------- --------- --------- ------- ---------= =20=20=20=20=20=20 >>> ether em1bridge bridge 0000002a link2=20= =20=20=20=20=20=20=20=20=20 >>>=20 >>> Name: ngctl2800 Type: socket ID: 00000036 Num hooks: 0 >>>=20 >>> Name: em1bridge Type: bridge ID: 0000002a Num hooks: 3 >>> Local hook Peer name Peer type Peer ID Peer hook= =20=20=20=20=20=20 >>> ---------- --------- --------- ------- ---------= =20=20=20=20=20=20 >>> link2 ngeth0 eiface 00000035 ether=20= =20=20=20=20=20=20=20=20=20 >>> link1 em1 ether 00000003 upper=20= =20=20=20=20=20=20=20=20=20 >>> link0 em1 ether 00000003 lower=20= =20=20=20=20=20=20=20=20=20 >>>=20 >>> Name: msk0 Type: ether ID: 00000001 Num hooks: 0 >>>=20 >>> The list of hooks for our bridge (em1bridge) is growing, and now we see= a new node (ngeth0) with one hook into that bridge. >>>=20 >>> =3D=3D=3D >>>=20 >>> ASIDE: If you wanted to script this=85 here's how you can test for an u= nused link: >>>=20 >>> Right now, we have link0, link1, and link2 for the bridge. If a link ex= ists for a bridge, the following command will return some info about the li= nk and return success (whereas if the link does not exist, the command will= return an error and exit with error-status): >>>=20 >>> dteske@scu0a.jbsd.vicor.com ~ $ sudo ngctl msg em1bridge: getstats 0 >>> Rec'd response "getstats" (4) from "[2a]:": >>> Args: >>> {} >>> dteske@scu0a.jbsd.vicor.com ~ $ sudo ngctl msg em1bridge: getstats 1 >>> Rec'd response "getstats" (4) from "[2a]:": >>> Args: >>> {} >>> dteske@scu0a.jbsd.vicor.com ~ $ sudo ngctl msg em1bridge: getstats 2 >>> Rec'd response "getstats" (4) from "[2a]:": >>> Args: >>> {} >>> dteske@scu0a.jbsd.vicor.com ~ $ sudo ngctl msg em1bridge: getstats 3 >>> ngctl: send msg: Socket is not connected >>> dteske@scu0a.jbsd.vicor.com ~ $ sudo ngctl msg em1bridge: getstats 4 >>> ngctl: send msg: Socket is not connected >>> dteske@scu0a.jbsd.vicor.com ~ $ sudo ngctl msg em1bridge: getstats 5 >>> ngctl: send msg: Socket is not connected >>>=20 >>> As you can see from the above output=85 we get errors for link3, link4,= and link5, because they don't exist. Naturally, testing $? exit status aft= er each of these commands would show how this can be scripted (HINT: throw = stdout/stderr to /dev/null and test $?). >>>=20 >>> =3D=3D=3D >>>=20 >>> At this point=85 you say "ifconfig": >>>=20 >>> dteske@oos0a.lbxrich.vicor.com ~ $ ifconfig >>> msk0: flags=3D8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500 >>> options=3Dc011a<TXCSUM,VLAN_MTU,VLAN_HWTAGGING,TSO4,VLAN_HWTSO,LI= NKSTATE> >>> ether xx:xx:xx:xx:xx:xx >>> media: Ethernet autoselect >>> em0: flags=3D8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu = 1500 >>> options=3D209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,= WOL_MAGIC> >>> ether xx:xx:xx:xx:xx:xx >>> inet xx.xx.xx.xx netmask 0xffffff80 broadcast xx.xx.xx.xx >>> media: Ethernet autoselect (1000baseT <full-duplex>) >>> status: active >>> em1: flags=3D8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metri= c 0 mtu 1500 >>> options=3D209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,= WOL_MAGIC> >>> ether xx:xx:xx:xx:xx:xx >>> media: Ethernet autoselect >>> status: no carrier >>> ipfw0: flags=3D8801<UP,SIMPLEX,MULTICAST> metric 0 mtu 65536 >>> lo0: flags=3D8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384 >>> options=3D3<RXCSUM,TXCSUM> >>> inet 127.0.0.1 netmask 0xff000000=20 >>> ngeth0: flags=3D8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500 >>> ether 00:00:00:00:00:00 >>>=20 >>> =3D=3D=3D >>>=20 >>> Ok, there are two problems with the network interface. >>>=20 >>> 1. It has a NULL MAC address (00:00:00:00:00:00). Good luck communicati= ng on the Internet (remember, we disabled setautosrc -- we intend to make u= p a MAC address that is unique). >>>=20 >>> 2. The name leaves something to be desired (if we're going to use this = with a vimage jail, it would be nice if the interface had the jail name in = it, so that when you do an "ngctl ls -l" or an "ngctl dot" =85 you're going= to see the jail name so it becomes clear which jails are hooked to which P= HY's through which bridges). >>>=20 >>> =3D=3D=3D >>>=20 >>> Let's tackle the easier one first=85 let's rename this new interface. >>>=20 >>> You and I already know that this interface that we want to rename is "n= geth0"=85 but you can actually extract the name from the link in the bridge. >>>=20 >>> dteske@scu0a.jbsd.vicor.com ~ $ sudo ngctl show -n em1bridge:link2 >>> Name: ngeth0 Type: eiface ID: 00000035 Num hooks: 1 >>>=20 >>>=20 >>> First, we rename it in netgraph (this does not affect the output of ifc= onfig -- and again, we do this to make "ngctl ls -l" and "ngctl dot" more p= alatable): >>>=20 >>> dteske@scu0a.jbsd.vicor.com ~ $ sudo ngctl name em1bridge:link2 ng0_myj= ail >>> dteske@scu0a.jbsd.vicor.com ~ $ sudo ngctl ls -l=20=20=20 >>> There are 6 total nodes: >>> Name: em0 Type: ether ID: 00000002 Num hooks: 0 >>>=20 >>> Name: em1 Type: ether ID: 00000003 Num hooks: 2 >>> Local hook Peer name Peer type Peer ID Peer hook= =20=20=20=20=20=20 >>> ---------- --------- --------- ------- ---------= =20=20=20=20=20=20 >>> upper em1bridge bridge 0000002a link1=20= =20=20=20=20=20=20=20=20=20 >>> lower em1bridge bridge 0000002a link0=20= =20=20=20=20=20=20=20=20=20 >>>=20 >>> Name: ngctl2843 Type: socket ID: 00000046 Num hooks: 0 >>>=20 >>> Name: ng0_myjail Type: eiface ID: 00000035 Num hooks: 1 >>> Local hook Peer name Peer type Peer ID Peer hook= =20=20=20=20=20=20 >>> ---------- --------- --------- ------- ---------= =20=20=20=20=20=20 >>> ether em1bridge bridge 0000002a link2=20= =20=20=20=20=20=20=20=20=20 >>>=20 >>> Name: em1bridge Type: bridge ID: 0000002a Num hooks: 3 >>> Local hook Peer name Peer type Peer ID Peer hook= =20=20=20=20=20=20 >>> ---------- --------- --------- ------- ---------= =20=20=20=20=20=20 >>> link2 ng0_myjail eiface 00000035 ether=20= =20=20=20=20=20=20=20=20=20 >>> link1 em1 ether 00000003 upper=20= =20=20=20=20=20=20=20=20=20 >>> link0 em1 ether 00000003 lower=20= =20=20=20=20=20=20=20=20=20 >>>=20 >>> Name: msk0 Type: ether ID: 00000001 Num hooks: 0 >>>=20 >>>=20 >>> Looking good. However, ifconfig hasn't changed=85 >>>=20 >>> dteske@scu0a.jbsd.vicor.com ~ $ ifconfig >>> ... >>> ngeth0: flags=3D8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500 >>> ether 00:00:00:00:00:00 >>>=20 >>> We want to rename the interface with ifconfig for a different reason. >>>=20 >>> We renamed the interface with netgraph earlier so that netgraph outputs= would be nice and easy to digest. >>>=20 >>> This time, we rename with ifconfig so that we can layer jails onto the = same rootdir. >>>=20 >>> The naming convention (which is the same naming convention I use for re= naming on the netgraph side) is: >>>=20 >>> ng#_name >>>=20 >>> The # always starts at zero for each jail where "name" is the name of t= he jail. >>>=20 >>> Again=85 I use this scheme so that I can layer jails onto the same root= -dir; /etc/rc.conf is then populated with things like: >>>=20 >>> ifconfig_ng0_myjail=3D... >>> ifconfig_ng0_myrouter=3D... >>> ifconfig_ng1_myrouter=3D... >>> ifconfig_ng0_anotherjail=3D... >>>=20 >>> So that when you say "service netif start" inside the vnet jail=85 it a= pplies the right settings. >>>=20 >>> So=85 we rename with ifconfig: >>>=20 >>> dteske@scu0a.jbsd.vicor.com ~ $ sudo ifconfig ngeth0 name ng0_myjail >>> dteske@scu0a.jbsd.vicor.com ~ $ ifconfig >>> ... >>> ng0_myjail: flags=3D8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500 >>> ether 00:00:00:00:00:00 >>>=20 >>> =3D=3D=3D >>>=20 >>> We're almost ready to shove this interface into a jail (which we haven'= t created yet). >>>=20 >>> But=85 we come back to that NULL MAC address. >>>=20 >>> NOTE: Forming your own MAC address, or even coming up with your own for= mula should not be taken lightly. >>>=20 >>> Here's a formula I use (which is based on several RFC's for MAC address= formation): >>>=20 >>> NOTE: In this context, ${_bridge} is em1 and $LINKNUM is 2 >>>=20 >>> # Set the MAC address of the new interface >>> # using a sensible algorithm to prevent >>> # conflicts on the network. >>> # >>> # MAC LP:LL:LB:BB:BB:BB >>> # P 2, 6, A, or E but usually 2 >>> # NOTE: Indicates "privately administered= " MAC >>> # L ng_bridge(4) link number (1-65535) >>> # B Same as bridged interface >>> # >>> _bridge_ether=3D$( ifconfig ${_bridge} et= her | >>> awk '/ether/{print $2}' ) >>> _ether_devid=3D"${_bridge_ether#??:??:?}" >>> n=3D$LINKNUM >>> _quad=3D$(($n & 15)) >>> case "${_quad}" in >>> 10) _quad=3Da;; 11) _quad=3Db;; 12) _quad= =3Dc;; >>> 13) _quad=3Dd;; 14) _quad=3De;; 15) _quad= =3Df;; >>> esac >>> _ether_devid=3D":${_quad}${_ether_devid}" >>> n=3D$(($n >> 4)) >>> _quad=3D$(($n & 15)) >>> case "${_quad}" in >>> 10) _quad=3Da;; 11) _quad=3Db;; 12) _quad= =3Dc;; >>> 13) _quad=3Dd;; 14) _quad=3De;; 15) _quad= =3Df;; >>> esac >>> _ether_devid=3D"${_quad}${_ether_devid}" >>> n=3D$(($n >> 4)) >>> _quad=3D$(($n & 15)) >>> case "${_quad}" in >>> 10) _quad=3Da;; 11) _quad=3Db;; 12) _quad= =3Dc;; >>> 13) _quad=3Dd;; 14) _quad=3De;; 15) _quad= =3Df;; >>> esac >>> _ether_devid=3D"2:${_quad}${_ether_devid}" >>> n=3D$(($n >> 4)) >>> _quad=3D$(($n & 15)) >>> case "${_quad}" in >>> 10) _quad=3Da;; 11) _quad=3Db;; 12) _quad= =3Dc;; >>> 13) _quad=3Dd;; 14) _quad=3De;; 15) _quad= =3Df;; >>> esac >>> _ether_devid=3D"${_quad}${_ether_devid}" >>> n=3D$(($n >> 4)) >>>=20 >>> After which=85 ${_ether_devid} holds a properly formed MAC address tha= t can (in every case I've tested) "get out". >>>=20 >>> Here's what I do to set it: >>>=20 >>> ifconfig ng0_myjail ether "${_ether_devid}" >>>=20 >>> Here's an example of how the MAC address was translated from the physic= al adapter to the ng_eiface(4) interface: >>>=20 >>> dteske@scu0a.jbsd.vicor.com ~ $ ifconfig em1; ifconfig ng0_myjail >>> em1: flags=3D8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metri= c 0 mtu 1500 >>> options=3D209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MA= GIC> >>> ether 00:0e:0c:ab:1b:76 >>> media: Ethernet autoselect >>> status: no carrier >>> ng0_myjail: flags=3D8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500 >>> ether 02:00:2c:ab:1b:76 >>>=20 >>> =3D=3D=3D >>>=20 >>> OK=85 we're now ready to shove that interface into a vimage jail. >>>=20 >>> But=85 >>>=20 >>> First we need a vimage jail. (this is not a tutorial on how to create, = manage, build, or do anything else with jails, vimage-jails, or vps-jails *= other* than give it a netgraph based interface) >>>=20 >>> I'm going to use my existing base machine as a fake jail (by pointing m= y jail's rootdir at "/"). >>>=20 >>> NOTE: Certain sysctl's have to be set appropriately before you fire up = the jail to make this vimage jail able to do "more" on the net. >>>=20 >>> dteske@scu0a.jbsd.vicor.com ~ $ sudo sysctl security.jail.set_hostname_= allowed=3D1 security.jail.sysvipc_allowed=3D1 security.jail.socket_unixipro= ute_only=3D1 >>> security.jail.set_hostname_allowed: 1 -> 1 >>> security.jail.sysvipc_allowed: 1 -> 1 >>> security.jail.socket_unixiproute_only: 0 -> 1 >>>=20 >>> NOTE: Unless you intend to reboot to restore the defaults later=85 you = might want to take down those previous values for restoration *after* we fi= re up the "vimage" jail. >>>=20 >>> dteske@scu0a.jbsd.vicor.com ~ $ sudo jail -i -c vnet name=3Dmyjail host= .hostname=3Dmyjail path=3D/ persist >>> 1 >>> dteske@scu0a.jbsd.vicor.com ~ $ jls >>> JID IP Address Hostname Path >>> 1 - myjail / >>>=20 >>> OK=85 we have a running jail (with the vnet property, making it a "vima= ge" jail -- which can accept network interfaces). >>>=20 >>> =3D=3D=3D >>>=20 >>> Right now our jail has no network interfaces (well, it has an unconfigu= red lo0). >>>=20 >>> dteske@scu0a.jbsd.vicor.com ~ $ sudo jexec myjail ifconfig >>> lo0: flags=3D8008<LOOPBACK,MULTICAST> metric 0 mtu 16384 >>> options=3D3<RXCSUM,TXCSUM> >>>=20 >>> So let's pass the netgraph created interface into the jail=85 >>>=20 >>> dteske@scu0a.jbsd.vicor.com ~ $ sudo ifconfig ng0_myjail vnet 1 >>> dteske@scu0a.jbsd.vicor.com ~ $ sudo jexec myjail ifconfig >>> lo0: flags=3D8008<LOOPBACK,MULTICAST> metric 0 mtu 16384 >>> options=3D3<RXCSUM,TXCSUM> >>> ng0_myjail: flags=3D8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500 >>> ether 02:00:2c:ab:1b:76 >>>=20 >>> Sweet! >>>=20 >>> =3D=3D=3D >>>=20 >>> Almost there=85 >>>=20 >>> Let's go into /etc/rc.conf, give it an IP, and start the network=85 >>>=20 >>> dteske@scu0a.jbsd.vicor.com ~ $ sudo sysrc ifconfig_ng0_myjail=3D"inet = 192.168.1.1 netmask 255.255.255.0" >>> /etc/rc.conf: ifconfig_ng0_myjail: -> inet 192.168.1.1 netmask 255.255= .255.0 >>> dteske@scu0a.jbsd.vicor.com ~ $ grep ng0 /etc/rc.conf >>> ifconfig_ng0_myjail=3D"inet 192.168.1.1 netmask 255.255.255.0" >>> dteske@scu0a.jbsd.vicor.com ~ $ sudo jexec myjail service netif start >>> Starting Network: lo0 ng0_myjail. >>> lo0: flags=3D8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384 >>> options=3D3<RXCSUM,TXCSUM> >>> inet 127.0.0.1 netmask 0xff000000=20 >>> ng0_myjail: flags=3D8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric= 0 mtu 1500 >>> ether 02:00:2c:ab:1b:76 >>> inet 192.168.1.1 netmask 0xffffff00 broadcast 192.168.1.255 >>>=20 >>> Now we're cookin' with gasoline! >>>=20 >>> =3D=3D=3D >>>=20 >>> Optionally go configure your base machine with an IP and have fun. >>=20 >> A quick conclusion=85 >>=20 >> Because we've built this all on top of netgraph=85 we can =85 graph it. >>=20 >> dteske@scu0a.jbsd.vicor.com ~ $ sudo ngctl dot | dot -Tsvg -o netgraph-s= cu0a.svg >>=20 >> I then uploaded the file to the web and here it is: >>=20 >> https://urldefense.proofpoint.com/v1/url?u=3Dhttp://druidbsd.sourceforge= .net/download/netgraph-scu0a.svg&k=3D%2FbkpAUdJWZuiTILCq%2FFnQg%3D%3D%0A&r= =3DMrjs6vR4%2Faj2Ns9%2FssHJjg%3D%3D%0A&m=3Dxe0XNgnKBiT9v8HzxwWwnNMOVN3YdEHm= TsIZfFoQA9Y%3D%0A&s=3D0a6ee990c4871ecc2a5b69879d11f65798fe70d8948fb9d4592bd= 915b1fd0199 >>=20 >> You should compare this directly to the output of "ngctl ls -l": >>=20 >> dteske@scu0a.jbsd.vicor.com ~ $ sudo ngctl ls -l >> There are 6 total nodes: >> Name: em0 Type: ether ID: 00000002 Num hooks: 0 >>=20 >> Name: em1 Type: ether ID: 00000003 Num hooks: 2 >> Local hook Peer name Peer type Peer ID Peer hook= =20=20=20=20=20=20 >> ---------- --------- --------- ------- ---------= =20=20=20=20=20=20 >> upper em1bridge bridge 0000002a link1=20=20= =20=20=20=20=20=20=20=20 >> lower em1bridge bridge 0000002a link0=20=20= =20=20=20=20=20=20=20=20 >>=20 >> Name: ng0_myjail Type: eiface ID: 00000035 Num hooks: 1 >> Local hook Peer name Peer type Peer ID Peer hook= =20=20=20=20=20=20 >> ---------- --------- --------- ------- ---------= =20=20=20=20=20=20 >> ether em1bridge bridge 0000002a link2=20=20= =20=20=20=20=20=20=20=20 >>=20 >> Name: em1bridge Type: bridge ID: 0000002a Num hooks: 3 >> Local hook Peer name Peer type Peer ID Peer hook= =20=20=20=20=20=20 >> ---------- --------- --------- ------- ---------= =20=20=20=20=20=20 >> link2 ng0_myjail eiface 00000035 ether=20=20= =20=20=20=20=20=20=20=20 >> link1 em1 ether 00000003 upper=20=20= =20=20=20=20=20=20=20=20 >> link0 em1 ether 00000003 lower=20=20= =20=20=20=20=20=20=20=20 >>=20 >> Name: ngctl8676 Type: socket ID: 00000049 Num hooks: 0 >>=20 >> Name: msk0 Type: ether ID: 00000001 Num hooks: 0 >>=20 >> You'll notice that when you graph the layout with "ngctl dot", the nodes= are rendered as boxes displaying their "Peer Name" up top, their "Peer Typ= e" in the lower-left, and their "Peer ID" in the bottom-right. >>=20 >> The edges from one node to another contains two octagons. These are the = "Local hook" and "Peer hook". >> --=20 >> Devin >>=20 >> _____________ >> The information contained in this message is proprietary and/or confiden= tial. If you are not the intended recipient, please: (i) delete the message= and all copies; (ii) do not disclose, distribute or use the message in any= manner; and (iii) notify the sender immediately. In addition, please be aw= are that any message addressed to our domain is subject to archiving and re= view by persons other than the intended recipient. Thank you. >=20 > Hello Devin, >=20 > If you live in the same city I will invite you for a couple of beer (I ha= ve to pay of course!) - I live in the Netherlands then let me know -=20 >=20 If I ever make it out there, I'll certainly shoot you an e-mail. > I followed your tutorial with the expected result: I can ping the em1 int= erface but I still have the same problems of before regarding external to i= nternal networks communication. >=20 > Please note that on my original host (no jail) the default gateway is 192= .168.1.254, that's important for what I am going to do. >=20 > I added a default route like : >=20 > route add default -interface ng0_myjail=20 >=20 Why not: jexec myjail route add -net default 192.168.1.254 ?? --=20 Devin > and then I try to : >=20 > jexec myjail ping 8.8.8.8 >=20 > I analyzed the wireshark capture and I can see that an ARP request for th= e 192.168.254 (with the MAC address of our virtual NIC as source, as expect= ed) go out my freebsd env - which is on a virtualbox - and it gets the answ= er but when I read the ARP table of the jail I can see it as incomplete. It= seems that packet going out but the answer is not received and it is confi= rmed when I try to sniff with tcpdump on my FREEBSD, I can't see any ARP re= quest going in both from my physical and virtual NIC, the same if I try : >=20 > ping 192.168.1.254 >=20 > Then I can see ping reply coming from wireshark but not from tcpdump on F= reeBDS. >=20 > Wireshark is attached on the host machine on the physical interface where= VB is attached in Bridged Mode, my original physical interface on FreeBSD = took the IP address from DHCP without problem then the problem is related t= o the jail. >=20 > do I try do accomplish a task which is not possible with JAIL or somethin= g is wrong in my configuration or worst, in my brain :P ? >=20 > Thanks in advance, > Pietro. >=20 >=20 >=20 >=20 >=20 >=20 _____________ The information contained in this message is proprietary and/or confidentia= l. If you are not the intended recipient, please: (i) delete the message an= d all copies; (ii) do not disclose, distribute or use the message in any ma= nner; and (iii) notify the sender immediately. In addition, please be aware= that any message addressed to our domain is subject to archiving and revie= w by persons other than the intended recipient. Thank you.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?13CA24D6AB415D428143D44749F57D7201F6EBFF>