Date: Mon, 3 Jun 2013 15:33:48 +0200 From: Pietro Paolini <pulsarpietro@aol.com> To: Devin Teske <dteske@freebsd.org> Cc: FreeBSD Questions <freebsd-questions@freebsd.org> Subject: Re: VIMAGE Message-ID: <93DE0F1F-FF8C-4A48-BE9B-70C0F0B84AE8@aol.com> In-Reply-To: <13CA24D6AB415D428143D44749F57D7201F6BCEB@ltcfiswmsgmb21> References: <DB90C1DC-66E4-4429-A888-44F4F9E4B98B@aol.com> <13CA24D6AB415D428143D44749F57D7201F68CBD@ltcfiswmsgmb21> <DA96E7A7-C419-4C73-A27B-D02BAB2CBE4E@aol.com> <13CA24D6AB415D428143D44749F57D7201F6B5F0@ltcfiswmsgmb21> <13CA24D6AB415D428143D44749F57D7201F6BCEB@ltcfiswmsgmb21>
next in thread | previous in thread | raw e-mail | index | archive | help
On Jun 1, 2013, at 5:26 AM, "Teske, Devin" <Devin.Teske@fisglobal.com> = wrote: >=20 > On May 31, 2013, at 3:05 PM, Teske, Devin wrote: >=20 >>=20 >> On May 31, 2013, at 1:48 AM, Pietro Paolini wrote: >>=20 >>>=20 >>> On May 30, 2013, at 6:25 PM, "Teske, Devin" = <Devin.Teske@fisglobal.com> wrote: >>>=20 >>>>=20 >>>> On May 30, 2013, at 3:35 AM, Pietro Paolini wrote: >>>>=20 >>>>> Hello all, >>>>>=20 >>>>> I am a new bye on the FreeBSD and I am looking at the VIMAGE = features experiencing some problems. >>>>> I added the options : >>>>> VIMAGE >>>>> if_bridge >>>>>=20 >>>>> and I removed >>>>> STCP >>>>>=20 >>>>> then I recompiled my kernel and install it. >>>>>=20 >>>>> After that, following this tutorial = http://imunes.tel.fer.hr/virtnet/eurobsdcon07_tutorial.pdf I tried the = "Exercise 2" which consist on=20 >>>>> the following commands: >>>>>=20 >>>>> vimage -c n1 >>>>> vimage -c n2 >>>>> ngctl mkpeer efface ether ether >>>>> ngctl mkpeer efface ether ether >>>>=20 >>>> Don't you just love autocorrect? (does the same thing to me=85 = turns "eiface" into "efface") >>>>=20 >>>>=20 >>>>> ngctl mkpeer em0: bridge lower link0 >>>>=20 >>>> Looks good. >>>>=20 >>>>=20 >>>>> ngctl name em0:lower bridge0 >>>>=20 >>>> I usually do my "connect" before the "name"=85 but shouldn't = matter. Should work all the same. >>>>=20 >>>>=20 >>>>> ngctl connect em0: bridge0: upper link1 >>>>=20 >>>> This looks wrong to me. >>>>=20 >>>> I'd expect: >>>>=20 >>>> ngctl connect em0: bridge0:lower upper link1 >>>>=20 >>>=20 >>>=20 >>> Many thanks for the answer Devin, >>> when I try to use that last command I receive: >>>=20 >>> ngctl connect em0: bridge0:lower upper link1 >>> ngctl: send msg: Invalid argument >>>=20 >>> What's wrong ? >>>=20 >>=20 >> Let's start from scratch on a freshly booted box=85 >>=20 >> dteske@scu0a.jbsd.vicor.com ~ $ sudo ngctl ls -l >> [sudo] Password: >> There are 4 total nodes: >> Name: em0 Type: ether ID: 00000002 Num = hooks: 0 >> Name: em1 Type: ether ID: 00000003 Num = hooks: 0 >> Name: ngctl1719 Type: socket ID: 00000004 Num = hooks: 0 >> Name: msk0 Type: ether ID: 00000001 Num = hooks: 0 >>=20 >> Ok=85 we have an "ether" type node for each of our physical adapters = (these are provided by ng_ether(4); you didn't have to do anything to = get these nodes). >>=20 >> We also have a single "socket" type node. This is the "ngctl" = connection to the netgraph subsystem (you can learn more by reading = ng_socket(4)). >>=20 >> Here's the corresponding hardware behind em0, em1, and msk0: >>=20 >> =3D=3D=3D >>=20 >> dteske@scu0a.jbsd.vicor.com ~ $ grep = '\(em\|e1000phy\|mskc\?\)[[:digit:]]' /var/run/dmesg.boot >> mskc0: <Marvell Yukon 88E8050 Gigabit Ethernet> port 0xdc00-0xdcff = mem 0xfcffc000-0xfcffffff irq 16 at device 0.0 on pci5 >> msk0: <Marvell Technology Group Ltd. Yukon EC Id 0xb6 Rev 0x02> on = mskc0 >> msk0: Ethernet address: xx:xx:xx:xx:xx:xx >> miibus0: <MII bus> on msk0 >> e1000phy0: <Marvell 88E1111 Gigabit PHY> PHY 0 on miibus0 >> e1000phy0: none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, = 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto >> mskc0: [ITHREAD] >> em0: <Intel(R) PRO/1000 Legacy Network Connection 1.0.3> port = 0xec80-0xecbf mem 0xfebe0000-0xfebfffff irq 16 at device 4.0 on pci7 >> em0: [FILTER] >> em0: Ethernet address: xx:xx:xx:xx:xx:xx >> em1: <Intel(R) PRO/1000 Legacy Network Connection 1.0.3> port = 0xec00-0xec3f mem 0xfeba0000-0xfebbffff,0xfeb80000-0xfeb9ffff irq 18 at = device 6.0 on pci7 >> em1: [FILTER] >> em1: Ethernet address: xx:xx:xx:xx:xx:xx >> em0: link state changed to UP >>=20 >> =3D=3D=3D >>=20 >> Next, let's make a bridge (think of it as a big software switch that = we're going to hook a bunch of interfaces; created, physical, or = otherwise). >>=20 >> Since I'm doing this over an SSH connection (a mistake I made earlier = today), I'm not going to touch em0 (the adapter my SSH connection is = using). Creating the bridge on an actively configured PHY will knock it = off the net. This is not to say you can't have an active configuration = on a bridged interface=85 just that the creation of the bridge = (something you should only do once each time you boot) will disrupt an = active connection. >>=20 >> So=85 >>=20 >> dteske@scu0a.jbsd.vicor.com ~ $ sudo ngctl mkpeer em1: bridge lower = link0 >>=20 >> NOTE: No output =3D=3D Success. >>=20 >> =3D=3D=3D >>=20 >> Now let's look at our handiwork=85 >>=20 >> dteske@scu0a.jbsd.vicor.com ~ $ sudo ngctl info em1:lower >> Name: <unnamed> Type: bridge ID: 00000007 Num = hooks: 1 >> Local hook Peer name Peer type Peer ID Peer = hook =20 >> ---------- --------- --------- ------- = --------- =20 >> link0 em1 ether 00000003 lower = =20 >>=20 >>=20 >> Ok, we see that the lower peer hook of the em1 ether-node goes off to = something named "link0". >>=20 >> To see where link0 is off-to=85 we need a full listing (back to = "ngctl ls -l"). >>=20 >>=20 >> dteske@scu0a.jbsd.vicor.com ~ $ sudo ngctl ls -l >> There are 5 total nodes: >> Name: <unnamed> Type: bridge ID: 00000007 Num = hooks: 1 >> Local hook Peer name Peer type Peer ID Peer = hook =20 >> ---------- --------- --------- ------- = --------- =20 >> link0 em1 ether 00000003 lower = =20 >> Name: em0 Type: ether ID: 00000002 Num = hooks: 0 >> Name: em1 Type: ether ID: 00000003 Num = hooks: 1 >> Local hook Peer name Peer type Peer ID Peer = hook =20 >> ---------- --------- --------- ------- = --------- =20 >> lower <unnamed> bridge 00000007 link0 = =20 >> Name: ngctl1762 Type: socket ID: 0000000b Num = hooks: 0 >> Name: msk0 Type: ether ID: 00000001 Num = hooks: 0 >>=20 >>=20 >> Matching "link0" in the first column to "link0" in the last-column, = we can see that this lower-link0 is to a bridge (with no name). >>=20 >> NOTE: When you're digesting the above output=85 it helps to imagine = whitespace in between the nodes with their respective hooks and other = nodes. Future pastes below will introduce such whitespace to make it = easier to read. >>=20 >> =3D=3D=3D >>=20 >> Right now, the only way to refer to the bridge is by way of = "em1:lower" (because we created the bridge right on the lower hook of = the em1 ether-node). >>=20 >> At this point, let's talk about naming. Giving our bridge a name is = entirely optional, but greatly clarifies the output of both "ngctl ls = -l" and "ngctl dot". >>=20 >> dteske@scu0a.jbsd.vicor.com ~ $ sudo ngctl name em1:lower em1bridge >> dteske@scu0a.jbsd.vicor.com ~ $ sudo ngctl ls -l >> There are 5 total nodes: >> Name: em0 Type: ether ID: 00000002 Num = hooks: 0 >>=20 >> Name: em1 Type: ether ID: 00000003 Num = hooks: 1 >> Local hook Peer name Peer type Peer ID Peer = hook =20 >> ---------- --------- --------- ------- = --------- =20 >> lower em1bridge bridge 00000007 link0 = =20 >>=20 >> Name: ngctl1831 Type: socket ID: 0000001a Num = hooks: 0 >>=20 >> Name: em1bridge Type: bridge ID: 00000007 Num = hooks: 1 >> Local hook Peer name Peer type Peer ID Peer = hook =20 >> ---------- --------- --------- ------- = --------- =20 >> link0 em1 ether 00000003 lower = =20 >>=20 >> Name: msk0 Type: ether ID: 00000001 Num = hooks: 0 >>=20 >> The new "em1bridge" name acts as an alias to "em1:lower" in future = ngctl commands. For example, "ngctl info em1:lower" and "ngctl info = em1bridge" can now be used interchangeably and produce the same results. >>=20 >> dteske@scu0a.jbsd.vicor.com ~ $ sudo ngctl info em1bridge: >> Name: em1bridge Type: bridge ID: 00000007 Num = hooks: 1 >> Local hook Peer name Peer type Peer ID Peer = hook =20 >> ---------- --------- --------- ------- = --------- =20 >> link0 em1 ether 00000003 lower = =20 >> dteske@scu0a.jbsd.vicor.com ~ $ sudo ngctl info em1:lower=20 >> Name: em1bridge Type: bridge ID: 00000007 Num = hooks: 1 >> Local hook Peer name Peer type Peer ID Peer = hook =20 >> ---------- --------- --------- ------- = --------- =20 >> link0 em1 ether 00000003 lower = =20 >>=20 >> =3D=3D=3D >>=20 >> We're not done with the bridge yet. Because we foresee the = possibility that it might be nice to be able to communicate with the = jail that we're going to later hook into this bridge=85 we should hook = the physical adapter's "upper" hook into the bridge. >>=20 >> If you don't do this, you won't be able to (for example) ping a jail = from the host where the host has only the PHY and the jail has only a = (yet uncreated) eiface. Regardless of the fact that the bridge uses the = PHY and the jail uses the bridge, to communicate with an IP that is = configured on the base host, you must hook the upper. >>=20 >> dteske@scu0a.jbsd.vicor.com ~ $ sudo ngctl connect em1: em1:lower = upper link1 >>=20 >> If you want to use the alias I set up earlier (of "em1bridge") that = works too (just don't forget the colon at the end of the alias): >>=20 >> dteske@scu0a.jbsd.vicor.com ~ $ sudo ngctl connect em1: em1bridge: = upper link1 >>=20 >> Here's the results: >>=20 >> dteske@scu0a.jbsd.vicor.com ~ $ sudo ngctl ls -l >> There are 5 total nodes: >> Name: em0 Type: ether ID: 00000002 Num = hooks: 0 >>=20 >> Name: em1 Type: ether ID: 00000003 Num = hooks: 2 >> Local hook Peer name Peer type Peer ID Peer = hook =20 >> ---------- --------- --------- ------- = --------- =20 >> upper em1bridge bridge 0000002a link1 = =20 >> lower em1bridge bridge 0000002a link0 = =20 >>=20 >> Name: ngctl1874 Type: socket ID: 00000030 Num = hooks: 0 >>=20 >> Name: em1bridge Type: bridge ID: 0000002a Num = hooks: 2 >> Local hook Peer name Peer type Peer ID Peer = hook =20 >> ---------- --------- --------- ------- = --------- =20 >> link1 em1 ether 00000003 upper = =20 >> link0 em1 ether 00000003 lower = =20 >>=20 >> Name: msk0 Type: ether ID: 00000001 Num = hooks: 0 >>=20 >>=20 >> NOTE: Some of the Peer ID's have changed, because I wanted to test = that the alias could be used; I used "sudo ngctl shutdown em1bridge:" = and re-executed up to the point where I connect the em1:upper into the = bridge=85 except this time using the alias of "em1bridge" instead of = "em1:lower" (indeed, you can use them interchangeably). >>=20 >> =3D=3D=3D >>=20 >> Ok=85 We've now done the hard part=85 which was to create and = configure a bridge that is usable by any new nodes we connect to it and = also (if you hooked the upper portion of em1 back into its own lower = which is acting as the bridge) the base machine can communicate with any = of the forth-coming jails (if on the same subnet at least). >>=20 >> There's an easy step that shouldn't be skipped though=85 >>=20 >> Before you can truly use this bridge with any other interfaces=85 >>=20 >> dteske@scu0a.jbsd.vicor.com ~ $ sudo ifconfig em1 up >> dteske@scu0a.jbsd.vicor.com ~ $ sudo ngctl msg em1: setpromisc 1 >> dteske@scu0a.jbsd.vicor.com ~ $ sudo ngctl msg em1: setautosrc 0 >>=20 >> A bridge cannot send packets out if the interface is down. >> A bridge cannot work properly without promiscuous mode. >> A bridge cannot send out packets for different addresses unless you = turn off "setautosrc" >>=20 >> =3D=3D=3D >>=20 >> Let's create our first virtual NIC and connect it to the bridge. >>=20 >> dteske@scu0a.jbsd.vicor.com ~ $ sudo ngctl mkpeer em1bridge: eiface = link2 ether >>=20 >> This command did two things. It created a new "eiface" node (see = ng_eiface(4)), and connected it to the bridge. >>=20 >> Let's have a look: >>=20 >> dteske@scu0a.jbsd.vicor.com ~ $ sudo ngctl ls -l >> There are 6 total nodes: >> Name: em0 Type: ether ID: 00000002 Num = hooks: 0 >>=20 >> Name: em1 Type: ether ID: 00000003 Num = hooks: 2 >> Local hook Peer name Peer type Peer ID Peer = hook =20 >> ---------- --------- --------- ------- = --------- =20 >> upper em1bridge bridge 0000002a link1 = =20 >> lower em1bridge bridge 0000002a link0 = =20 >>=20 >> Name: ngeth0 Type: eiface ID: 00000035 Num = hooks: 1 >> Local hook Peer name Peer type Peer ID Peer = hook =20 >> ---------- --------- --------- ------- = --------- =20 >> ether em1bridge bridge 0000002a link2 = =20 >>=20 >> Name: ngctl2800 Type: socket ID: 00000036 Num = hooks: 0 >>=20 >> Name: em1bridge Type: bridge ID: 0000002a Num = hooks: 3 >> Local hook Peer name Peer type Peer ID Peer = hook =20 >> ---------- --------- --------- ------- = --------- =20 >> link2 ngeth0 eiface 00000035 ether = =20 >> link1 em1 ether 00000003 upper = =20 >> link0 em1 ether 00000003 lower = =20 >>=20 >> Name: msk0 Type: ether ID: 00000001 Num = hooks: 0 >>=20 >> The list of hooks for our bridge (em1bridge) is growing, and now we = see a new node (ngeth0) with one hook into that bridge. >>=20 >> =3D=3D=3D >>=20 >> ASIDE: If you wanted to script this=85 here's how you can test for an = unused link: >>=20 >> Right now, we have link0, link1, and link2 for the bridge. If a link = exists for a bridge, the following command will return some info about = the link and return success (whereas if the link does not exist, the = command will return an error and exit with error-status): >>=20 >> dteske@scu0a.jbsd.vicor.com ~ $ sudo ngctl msg em1bridge: getstats 0 >> Rec'd response "getstats" (4) from "[2a]:": >> Args: >> {} >> dteske@scu0a.jbsd.vicor.com ~ $ sudo ngctl msg em1bridge: getstats 1 >> Rec'd response "getstats" (4) from "[2a]:": >> Args: >> {} >> dteske@scu0a.jbsd.vicor.com ~ $ sudo ngctl msg em1bridge: getstats 2 >> Rec'd response "getstats" (4) from "[2a]:": >> Args: >> {} >> dteske@scu0a.jbsd.vicor.com ~ $ sudo ngctl msg em1bridge: getstats 3 >> ngctl: send msg: Socket is not connected >> dteske@scu0a.jbsd.vicor.com ~ $ sudo ngctl msg em1bridge: getstats 4 >> ngctl: send msg: Socket is not connected >> dteske@scu0a.jbsd.vicor.com ~ $ sudo ngctl msg em1bridge: getstats 5 >> ngctl: send msg: Socket is not connected >>=20 >> As you can see from the above output=85 we get errors for link3, = link4, and link5, because they don't exist. Naturally, testing $? exit = status after each of these commands would show how this can be scripted = (HINT: throw stdout/stderr to /dev/null and test $?). >>=20 >> =3D=3D=3D >>=20 >> At this point=85 you say "ifconfig": >>=20 >> dteske@oos0a.lbxrich.vicor.com ~ $ ifconfig >> msk0: flags=3D8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500 >> = options=3Dc011a<TXCSUM,VLAN_MTU,VLAN_HWTAGGING,TSO4,VLAN_HWTSO,LINKSTATE> >> ether xx:xx:xx:xx:xx:xx >> media: Ethernet autoselect >> em0: flags=3D8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 = mtu 1500 >> = options=3D209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC= > >> ether xx:xx:xx:xx:xx:xx >> inet xx.xx.xx.xx netmask 0xffffff80 broadcast xx.xx.xx.xx >> media: Ethernet autoselect (1000baseT <full-duplex>) >> status: active >> em1: flags=3D8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> = metric 0 mtu 1500 >> = options=3D209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC= > >> ether xx:xx:xx:xx:xx:xx >> media: Ethernet autoselect >> status: no carrier >> ipfw0: flags=3D8801<UP,SIMPLEX,MULTICAST> metric 0 mtu 65536 >> lo0: flags=3D8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384 >> options=3D3<RXCSUM,TXCSUM> >> inet 127.0.0.1 netmask 0xff000000=20 >> ngeth0: flags=3D8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500 >> ether 00:00:00:00:00:00 >>=20 >> =3D=3D=3D >>=20 >> Ok, there are two problems with the network interface. >>=20 >> 1. It has a NULL MAC address (00:00:00:00:00:00). Good luck = communicating on the Internet (remember, we disabled setautosrc -- we = intend to make up a MAC address that is unique). >>=20 >> 2. The name leaves something to be desired (if we're going to use = this with a vimage jail, it would be nice if the interface had the jail = name in it, so that when you do an "ngctl ls -l" or an "ngctl dot" =85 = you're going to see the jail name so it becomes clear which jails are = hooked to which PHY's through which bridges). >>=20 >> =3D=3D=3D >>=20 >> Let's tackle the easier one first=85 let's rename this new interface. >>=20 >> You and I already know that this interface that we want to rename is = "ngeth0"=85 but you can actually extract the name from the link in the = bridge. >>=20 >> dteske@scu0a.jbsd.vicor.com ~ $ sudo ngctl show -n em1bridge:link2 >> Name: ngeth0 Type: eiface ID: 00000035 Num = hooks: 1 >>=20 >>=20 >> First, we rename it in netgraph (this does not affect the output of = ifconfig -- and again, we do this to make "ngctl ls -l" and "ngctl dot" = more palatable): >>=20 >> dteske@scu0a.jbsd.vicor.com ~ $ sudo ngctl name em1bridge:link2 = ng0_myjail >> dteske@scu0a.jbsd.vicor.com ~ $ sudo ngctl ls -l =20 >> There are 6 total nodes: >> Name: em0 Type: ether ID: 00000002 Num = hooks: 0 >>=20 >> Name: em1 Type: ether ID: 00000003 Num = hooks: 2 >> Local hook Peer name Peer type Peer ID Peer = hook =20 >> ---------- --------- --------- ------- = --------- =20 >> upper em1bridge bridge 0000002a link1 = =20 >> lower em1bridge bridge 0000002a link0 = =20 >>=20 >> Name: ngctl2843 Type: socket ID: 00000046 Num = hooks: 0 >>=20 >> Name: ng0_myjail Type: eiface ID: 00000035 Num = hooks: 1 >> Local hook Peer name Peer type Peer ID Peer = hook =20 >> ---------- --------- --------- ------- = --------- =20 >> ether em1bridge bridge 0000002a link2 = =20 >>=20 >> Name: em1bridge Type: bridge ID: 0000002a Num = hooks: 3 >> Local hook Peer name Peer type Peer ID Peer = hook =20 >> ---------- --------- --------- ------- = --------- =20 >> link2 ng0_myjail eiface 00000035 ether = =20 >> link1 em1 ether 00000003 upper = =20 >> link0 em1 ether 00000003 lower = =20 >>=20 >> Name: msk0 Type: ether ID: 00000001 Num = hooks: 0 >>=20 >>=20 >> Looking good. However, ifconfig hasn't changed=85 >>=20 >> dteske@scu0a.jbsd.vicor.com ~ $ ifconfig >> ... >> ngeth0: flags=3D8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500 >> ether 00:00:00:00:00:00 >>=20 >> We want to rename the interface with ifconfig for a different reason. >>=20 >> We renamed the interface with netgraph earlier so that netgraph = outputs would be nice and easy to digest. >>=20 >> This time, we rename with ifconfig so that we can layer jails onto = the same rootdir. >>=20 >> The naming convention (which is the same naming convention I use for = renaming on the netgraph side) is: >>=20 >> ng#_name >>=20 >> The # always starts at zero for each jail where "name" is the name of = the jail. >>=20 >> Again=85 I use this scheme so that I can layer jails onto the same = root-dir; /etc/rc.conf is then populated with things like: >>=20 >> ifconfig_ng0_myjail=3D... >> ifconfig_ng0_myrouter=3D... >> ifconfig_ng1_myrouter=3D... >> ifconfig_ng0_anotherjail=3D... >>=20 >> So that when you say "service netif start" inside the vnet jail=85 it = applies the right settings. >>=20 >> So=85 we rename with ifconfig: >>=20 >> dteske@scu0a.jbsd.vicor.com ~ $ sudo ifconfig ngeth0 name ng0_myjail >> dteske@scu0a.jbsd.vicor.com ~ $ ifconfig >> ... >> ng0_myjail: flags=3D8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu = 1500 >> ether 00:00:00:00:00:00 >>=20 >> =3D=3D=3D >>=20 >> We're almost ready to shove this interface into a jail (which we = haven't created yet). >>=20 >> But=85 we come back to that NULL MAC address. >>=20 >> NOTE: Forming your own MAC address, or even coming up with your own = formula should not be taken lightly. >>=20 >> Here's a formula I use (which is based on several RFC's for MAC = address formation): >>=20 >> NOTE: In this context, ${_bridge} is em1 and $LINKNUM is 2 >>=20 >> # Set the MAC address of the new = interface >> # using a sensible algorithm to = prevent >> # conflicts on the network. >> # >> # MAC LP:LL:LB:BB:BB:BB >> # P 2, 6, A, or E but usually 2 >> # NOTE: Indicates "privately = administered" MAC >> # L ng_bridge(4) link number = (1-65535) >> # B Same as bridged interface >> # >> _bridge_ether=3D$( ifconfig ${_bridge} = ether | >> awk '/ether/{print $2}' ) >> = _ether_devid=3D"${_bridge_ether#??:??:?}" >> n=3D$LINKNUM >> _quad=3D$(($n & 15)) >> case "${_quad}" in >> 10) _quad=3Da;; 11) _quad=3Db;; 12) = _quad=3Dc;; >> 13) _quad=3Dd;; 14) _quad=3De;; 15) = _quad=3Df;; >> esac >> = _ether_devid=3D":${_quad}${_ether_devid}" >> n=3D$(($n >> 4)) >> _quad=3D$(($n & 15)) >> case "${_quad}" in >> 10) _quad=3Da;; 11) _quad=3Db;; 12) = _quad=3Dc;; >> 13) _quad=3Dd;; 14) _quad=3De;; 15) = _quad=3Df;; >> esac >> _ether_devid=3D"${_quad}${_ether_devid}"= >> n=3D$(($n >> 4)) >> _quad=3D$(($n & 15)) >> case "${_quad}" in >> 10) _quad=3Da;; 11) _quad=3Db;; 12) = _quad=3Dc;; >> 13) _quad=3Dd;; 14) _quad=3De;; 15) = _quad=3Df;; >> esac >> = _ether_devid=3D"2:${_quad}${_ether_devid}" >> n=3D$(($n >> 4)) >> _quad=3D$(($n & 15)) >> case "${_quad}" in >> 10) _quad=3Da;; 11) _quad=3Db;; 12) = _quad=3Dc;; >> 13) _quad=3Dd;; 14) _quad=3De;; 15) = _quad=3Df;; >> esac >> _ether_devid=3D"${_quad}${_ether_devid}"= >> n=3D$(($n >> 4)) >>=20 >> After which=85 ${_ether_devid} holds a properly formed MAC address = that can (in every case I've tested) "get out". >>=20 >> Here's what I do to set it: >>=20 >> ifconfig ng0_myjail ether "${_ether_devid}" >>=20 >> Here's an example of how the MAC address was translated from the = physical adapter to the ng_eiface(4) interface: >>=20 >> dteske@scu0a.jbsd.vicor.com ~ $ ifconfig em1; ifconfig ng0_myjail >> em1: flags=3D8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> = metric 0 mtu 1500 >> = options=3D209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC= > >> ether 00:0e:0c:ab:1b:76 >> media: Ethernet autoselect >> status: no carrier >> ng0_myjail: flags=3D8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu = 1500 >> ether 02:00:2c:ab:1b:76 >>=20 >> =3D=3D=3D >>=20 >> OK=85 we're now ready to shove that interface into a vimage jail. >>=20 >> But=85 >>=20 >> First we need a vimage jail. (this is not a tutorial on how to = create, manage, build, or do anything else with jails, vimage-jails, or = vps-jails *other* than give it a netgraph based interface) >>=20 >> I'm going to use my existing base machine as a fake jail (by pointing = my jail's rootdir at "/"). >>=20 >> NOTE: Certain sysctl's have to be set appropriately before you fire = up the jail to make this vimage jail able to do "more" on the net. >>=20 >> dteske@scu0a.jbsd.vicor.com ~ $ sudo sysctl = security.jail.set_hostname_allowed=3D1 security.jail.sysvipc_allowed=3D1 = security.jail.socket_unixiproute_only=3D1 >> security.jail.set_hostname_allowed: 1 -> 1 >> security.jail.sysvipc_allowed: 1 -> 1 >> security.jail.socket_unixiproute_only: 0 -> 1 >>=20 >> NOTE: Unless you intend to reboot to restore the defaults later=85 = you might want to take down those previous values for restoration = *after* we fire up the "vimage" jail. >>=20 >> dteske@scu0a.jbsd.vicor.com ~ $ sudo jail -i -c vnet name=3Dmyjail = host.hostname=3Dmyjail path=3D/ persist >> 1 >> dteske@scu0a.jbsd.vicor.com ~ $ jls >> JID IP Address Hostname Path >> 1 - myjail / >>=20 >> OK=85 we have a running jail (with the vnet property, making it a = "vimage" jail -- which can accept network interfaces). >>=20 >> =3D=3D=3D >>=20 >> Right now our jail has no network interfaces (well, it has an = unconfigured lo0). >>=20 >> dteske@scu0a.jbsd.vicor.com ~ $ sudo jexec myjail ifconfig >> lo0: flags=3D8008<LOOPBACK,MULTICAST> metric 0 mtu 16384 >> options=3D3<RXCSUM,TXCSUM> >>=20 >> So let's pass the netgraph created interface into the jail=85 >>=20 >> dteske@scu0a.jbsd.vicor.com ~ $ sudo ifconfig ng0_myjail vnet 1 >> dteske@scu0a.jbsd.vicor.com ~ $ sudo jexec myjail ifconfig >> lo0: flags=3D8008<LOOPBACK,MULTICAST> metric 0 mtu 16384 >> options=3D3<RXCSUM,TXCSUM> >> ng0_myjail: flags=3D8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu = 1500 >> ether 02:00:2c:ab:1b:76 >>=20 >> Sweet! >>=20 >> =3D=3D=3D >>=20 >> Almost there=85 >>=20 >> Let's go into /etc/rc.conf, give it an IP, and start the network=85 >>=20 >> dteske@scu0a.jbsd.vicor.com ~ $ sudo sysrc ifconfig_ng0_myjail=3D"inet = 192.168.1.1 netmask 255.255.255.0" >> /etc/rc.conf: ifconfig_ng0_myjail: -> inet 192.168.1.1 netmask = 255.255.255.0 >> dteske@scu0a.jbsd.vicor.com ~ $ grep ng0 /etc/rc.conf >> ifconfig_ng0_myjail=3D"inet 192.168.1.1 netmask 255.255.255.0" >> dteske@scu0a.jbsd.vicor.com ~ $ sudo jexec myjail service netif start >> Starting Network: lo0 ng0_myjail. >> lo0: flags=3D8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384 >> options=3D3<RXCSUM,TXCSUM> >> inet 127.0.0.1 netmask 0xff000000=20 >> ng0_myjail: flags=3D8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> = metric 0 mtu 1500 >> ether 02:00:2c:ab:1b:76 >> inet 192.168.1.1 netmask 0xffffff00 broadcast 192.168.1.255 >>=20 >> Now we're cookin' with gasoline! >>=20 >> =3D=3D=3D >>=20 >> Optionally go configure your base machine with an IP and have fun. >=20 > A quick conclusion=85 >=20 > Because we've built this all on top of netgraph=85 we can =85 graph = it. >=20 > dteske@scu0a.jbsd.vicor.com ~ $ sudo ngctl dot | dot -Tsvg -o = netgraph-scu0a.svg >=20 > I then uploaded the file to the web and here it is: >=20 > http://druidbsd.sourceforge.net/download/netgraph-scu0a.svg >=20 > You should compare this directly to the output of "ngctl ls -l": >=20 > dteske@scu0a.jbsd.vicor.com ~ $ sudo ngctl ls -l > There are 6 total nodes: > Name: em0 Type: ether ID: 00000002 Num hooks: = 0 >=20 > Name: em1 Type: ether ID: 00000003 Num hooks: = 2 > Local hook Peer name Peer type Peer ID Peer = hook =20 > ---------- --------- --------- ------- = --------- =20 > upper em1bridge bridge 0000002a link1 = =20 > lower em1bridge bridge 0000002a link0 = =20 >=20 > Name: ng0_myjail Type: eiface ID: 00000035 Num hooks: = 1 > Local hook Peer name Peer type Peer ID Peer = hook =20 > ---------- --------- --------- ------- = --------- =20 > ether em1bridge bridge 0000002a link2 = =20 >=20 > Name: em1bridge Type: bridge ID: 0000002a Num hooks: = 3 > Local hook Peer name Peer type Peer ID Peer = hook =20 > ---------- --------- --------- ------- = --------- =20 > link2 ng0_myjail eiface 00000035 ether = =20 > link1 em1 ether 00000003 upper = =20 > link0 em1 ether 00000003 lower = =20 >=20 > Name: ngctl8676 Type: socket ID: 00000049 Num hooks: = 0 >=20 > Name: msk0 Type: ether ID: 00000001 Num hooks: = 0 >=20 > You'll notice that when you graph the layout with "ngctl dot", the = nodes are rendered as boxes displaying their "Peer Name" up top, their = "Peer Type" in the lower-left, and their "Peer ID" in the bottom-right. >=20 > The edges from one node to another contains two octagons. These are = the "Local hook" and "Peer hook". > --=20 > Devin >=20 > _____________ > The information contained in this message is proprietary and/or = confidential. If you are not the intended recipient, please: (i) delete = the message and all copies; (ii) do not disclose, distribute or use the = message in any manner; and (iii) notify the sender immediately. In = addition, please be aware that any message addressed to our domain is = subject to archiving and review by persons other than the intended = recipient. Thank you. Hello Devin, If you live in the same city I will invite you for a couple of beer (I = have to pay of course!) - I live in the Netherlands then let me know -=20= I followed your tutorial with the expected result: I can ping the em1 = interface but I still have the same problems of before regarding = external to internal networks communication. Please note that on my original host (no jail) the default gateway is = 192.168.1.254, that's important for what I am going to do. I added a default route like : route add default -interface ng0_myjail=20 and then I try to : jexec myjail ping 8.8.8.8 I analyzed the wireshark capture and I can see that an ARP request for = the 192.168.254 (with the MAC address of our virtual NIC as source, as = expected) go out my freebsd env - which is on a virtualbox - and it gets = the answer but when I read the ARP table of the jail I can see it as = incomplete. It seems that packet going out but the answer is not = received and it is confirmed when I try to sniff with tcpdump on my = FREEBSD, I can't see any ARP request going in both from my physical and = virtual NIC, the same if I try : ping 192.168.1.254 Then I can see ping reply coming from wireshark but not from tcpdump on = FreeBDS. Wireshark is attached on the host machine on the physical interface = where VB is attached in Bridged Mode, my original physical interface on = FreeBSD took the IP address from DHCP without problem then the problem = is related to the jail. do I try do accomplish a task which is not possible with JAIL or = something is wrong in my configuration or worst, in my brain :P ? Thanks in advance, Pietro.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?93DE0F1F-FF8C-4A48-BE9B-70C0F0B84AE8>