Skip site navigation (1)Skip section navigation (2)
Date:      Mon, 3 Jun 2013 15:33:48 +0200
From:      Pietro Paolini <pulsarpietro@aol.com>
To:        Devin Teske <dteske@freebsd.org>
Cc:        FreeBSD Questions <freebsd-questions@freebsd.org>
Subject:   Re: VIMAGE
Message-ID:  <93DE0F1F-FF8C-4A48-BE9B-70C0F0B84AE8@aol.com>
In-Reply-To: <13CA24D6AB415D428143D44749F57D7201F6BCEB@ltcfiswmsgmb21>
References:  <DB90C1DC-66E4-4429-A888-44F4F9E4B98B@aol.com> <13CA24D6AB415D428143D44749F57D7201F68CBD@ltcfiswmsgmb21> <DA96E7A7-C419-4C73-A27B-D02BAB2CBE4E@aol.com> <13CA24D6AB415D428143D44749F57D7201F6B5F0@ltcfiswmsgmb21> <13CA24D6AB415D428143D44749F57D7201F6BCEB@ltcfiswmsgmb21>

Next in thread | Previous in thread | Raw E-Mail | Index | Archive | Help

On Jun 1, 2013, at 5:26 AM, "Teske, Devin" <Devin.Teske@fisglobal.com> =
wrote:

>=20
> On May 31, 2013, at 3:05 PM, Teske, Devin wrote:
>=20
>>=20
>> On May 31, 2013, at 1:48 AM, Pietro Paolini wrote:
>>=20
>>>=20
>>> On May 30, 2013, at 6:25 PM, "Teske, Devin" =
<Devin.Teske@fisglobal.com> wrote:
>>>=20
>>>>=20
>>>> On May 30, 2013, at 3:35 AM, Pietro Paolini wrote:
>>>>=20
>>>>> Hello all,
>>>>>=20
>>>>> I am a new bye on the FreeBSD and I am looking at the VIMAGE =
features experiencing some problems.
>>>>> I added the options :
>>>>> VIMAGE
>>>>> if_bridge
>>>>>=20
>>>>> and I removed
>>>>> STCP
>>>>>=20
>>>>> then I recompiled my kernel and install it.
>>>>>=20
>>>>> After that, following this tutorial =
http://imunes.tel.fer.hr/virtnet/eurobsdcon07_tutorial.pdf I tried the =
"Exercise 2" which consist on=20
>>>>> the following commands:
>>>>>=20
>>>>> vimage -c n1
>>>>> vimage -c n2
>>>>> ngctl mkpeer efface ether ether
>>>>> ngctl mkpeer efface ether ether
>>>>=20
>>>> Don't you just love autocorrect? (does the same thing to me=85 =
turns "eiface" into "efface")
>>>>=20
>>>>=20
>>>>> ngctl mkpeer em0: bridge lower link0
>>>>=20
>>>> Looks good.
>>>>=20
>>>>=20
>>>>> ngctl name em0:lower bridge0
>>>>=20
>>>> I usually do my "connect" before the "name"=85 but shouldn't =
matter. Should work all the same.
>>>>=20
>>>>=20
>>>>> ngctl connect em0: bridge0: upper link1
>>>>=20
>>>> This looks wrong to me.
>>>>=20
>>>> I'd expect:
>>>>=20
>>>> ngctl connect em0: bridge0:lower upper link1
>>>>=20
>>>=20
>>>=20
>>> Many thanks for the answer Devin,
>>> when I try to use that last command I receive:
>>>=20
>>> ngctl connect em0: bridge0:lower upper link1
>>> ngctl: send msg: Invalid argument
>>>=20
>>> What's wrong ?
>>>=20
>>=20
>> Let's start from scratch on a freshly booted box=85
>>=20
>> dteske@scu0a.jbsd.vicor.com ~ $ sudo ngctl ls -l
>> [sudo] Password:
>> There are 4 total nodes:
>>  Name: em0             Type: ether           ID: 00000002   Num =
hooks: 0
>>  Name: em1             Type: ether           ID: 00000003   Num =
hooks: 0
>>  Name: ngctl1719       Type: socket          ID: 00000004   Num =
hooks: 0
>>  Name: msk0            Type: ether           ID: 00000001   Num =
hooks: 0
>>=20
>> Ok=85 we have an "ether" type node for each of our physical adapters =
(these are provided by ng_ether(4); you didn't have to do anything to =
get these nodes).
>>=20
>> We also have a single "socket" type node. This is the "ngctl" =
connection to the netgraph subsystem (you can learn more by reading =
ng_socket(4)).
>>=20
>> Here's the corresponding hardware behind em0, em1, and msk0:
>>=20
>> =3D=3D=3D
>>=20
>> dteske@scu0a.jbsd.vicor.com ~ $ grep =
'\(em\|e1000phy\|mskc\?\)[[:digit:]]' /var/run/dmesg.boot
>> mskc0: <Marvell Yukon 88E8050 Gigabit Ethernet> port 0xdc00-0xdcff =
mem 0xfcffc000-0xfcffffff irq 16 at device 0.0 on pci5
>> msk0: <Marvell Technology Group Ltd. Yukon EC Id 0xb6 Rev 0x02> on =
mskc0
>> msk0: Ethernet address: xx:xx:xx:xx:xx:xx
>> miibus0: <MII bus> on msk0
>> e1000phy0: <Marvell 88E1111 Gigabit PHY> PHY 0 on miibus0
>> e1000phy0:  none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, =
1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto
>> mskc0: [ITHREAD]
>> em0: <Intel(R) PRO/1000 Legacy Network Connection 1.0.3> port =
0xec80-0xecbf mem 0xfebe0000-0xfebfffff irq 16 at device 4.0 on pci7
>> em0: [FILTER]
>> em0: Ethernet address: xx:xx:xx:xx:xx:xx
>> em1: <Intel(R) PRO/1000 Legacy Network Connection 1.0.3> port =
0xec00-0xec3f mem 0xfeba0000-0xfebbffff,0xfeb80000-0xfeb9ffff irq 18 at =
device 6.0 on pci7
>> em1: [FILTER]
>> em1: Ethernet address: xx:xx:xx:xx:xx:xx
>> em0: link state changed to UP
>>=20
>> =3D=3D=3D
>>=20
>> Next, let's make a bridge (think of it as a big software switch that =
we're going to hook a bunch of interfaces; created, physical, or =
otherwise).
>>=20
>> Since I'm doing this over an SSH connection (a mistake I made earlier =
today), I'm not going to touch em0 (the adapter my SSH connection is =
using). Creating the bridge on an actively configured PHY will knock it =
off the net. This is not to say you can't have an active configuration =
on a bridged interface=85 just that the creation of the bridge =
(something you should only do once each time you boot) will disrupt an =
active connection.
>>=20
>> So=85
>>=20
>> dteske@scu0a.jbsd.vicor.com ~ $ sudo ngctl mkpeer em1: bridge lower =
link0
>>=20
>> NOTE: No output =3D=3D Success.
>>=20
>> =3D=3D=3D
>>=20
>> Now let's look at our handiwork=85
>>=20
>> dteske@scu0a.jbsd.vicor.com ~ $ sudo ngctl info em1:lower
>>  Name: <unnamed>       Type: bridge          ID: 00000007   Num =
hooks: 1
>>  Local hook      Peer name       Peer type    Peer ID         Peer =
hook     =20
>>  ----------      ---------       ---------    -------         =
---------     =20
>>  link0           em1             ether        00000003        lower   =
      =20
>>=20
>>=20
>> Ok, we see that the lower peer hook of the em1 ether-node goes off to =
something named "link0".
>>=20
>> To see where link0 is off-to=85 we need a full listing (back to =
"ngctl ls -l").
>>=20
>>=20
>> dteske@scu0a.jbsd.vicor.com ~ $ sudo ngctl ls -l
>> There are 5 total nodes:
>>  Name: <unnamed>       Type: bridge          ID: 00000007   Num =
hooks: 1
>>  Local hook      Peer name       Peer type    Peer ID         Peer =
hook     =20
>>  ----------      ---------       ---------    -------         =
---------     =20
>>  link0           em1             ether        00000003        lower   =
      =20
>>  Name: em0             Type: ether           ID: 00000002   Num =
hooks: 0
>>  Name: em1             Type: ether           ID: 00000003   Num =
hooks: 1
>>  Local hook      Peer name       Peer type    Peer ID         Peer =
hook     =20
>>  ----------      ---------       ---------    -------         =
---------     =20
>>  lower           <unnamed>       bridge       00000007        link0   =
      =20
>>  Name: ngctl1762       Type: socket          ID: 0000000b   Num =
hooks: 0
>>  Name: msk0            Type: ether           ID: 00000001   Num =
hooks: 0
>>=20
>>=20
>> Matching "link0" in the first column to "link0" in the last-column, =
we can see that this lower-link0 is to a bridge (with no name).
>>=20
>> NOTE: When you're digesting the above output=85 it helps to imagine =
whitespace in between the nodes with their respective hooks and other =
nodes. Future pastes below will introduce such whitespace to make it =
easier to read.
>>=20
>> =3D=3D=3D
>>=20
>> Right now, the only way to refer to the bridge is by way of =
"em1:lower" (because we created the bridge right on the lower hook of =
the em1 ether-node).
>>=20
>> At this point, let's talk about naming. Giving our bridge a name is =
entirely optional, but greatly clarifies the output of both "ngctl ls =
-l" and "ngctl dot".
>>=20
>> dteske@scu0a.jbsd.vicor.com ~ $ sudo ngctl name em1:lower em1bridge
>> dteske@scu0a.jbsd.vicor.com ~ $ sudo ngctl ls -l
>> There are 5 total nodes:
>>  Name: em0             Type: ether           ID: 00000002   Num =
hooks: 0
>>=20
>>  Name: em1             Type: ether           ID: 00000003   Num =
hooks: 1
>>  Local hook      Peer name       Peer type    Peer ID         Peer =
hook     =20
>>  ----------      ---------       ---------    -------         =
---------     =20
>>  lower           em1bridge       bridge       00000007        link0   =
      =20
>>=20
>>  Name: ngctl1831       Type: socket          ID: 0000001a   Num =
hooks: 0
>>=20
>>  Name: em1bridge       Type: bridge          ID: 00000007   Num =
hooks: 1
>>  Local hook      Peer name       Peer type    Peer ID         Peer =
hook     =20
>>  ----------      ---------       ---------    -------         =
---------     =20
>>  link0           em1             ether        00000003        lower   =
      =20
>>=20
>>  Name: msk0            Type: ether           ID: 00000001   Num =
hooks: 0
>>=20
>> The new "em1bridge" name acts as an alias to "em1:lower" in future =
ngctl commands. For example, "ngctl info em1:lower" and "ngctl info =
em1bridge" can now be used interchangeably and produce the same results.
>>=20
>> dteske@scu0a.jbsd.vicor.com ~ $ sudo ngctl info em1bridge:
>>  Name: em1bridge       Type: bridge          ID: 00000007   Num =
hooks: 1
>>  Local hook      Peer name       Peer type    Peer ID         Peer =
hook     =20
>>  ----------      ---------       ---------    -------         =
---------     =20
>>  link0           em1             ether        00000003        lower   =
      =20
>> dteske@scu0a.jbsd.vicor.com ~ $ sudo ngctl info em1:lower=20
>>  Name: em1bridge       Type: bridge          ID: 00000007   Num =
hooks: 1
>>  Local hook      Peer name       Peer type    Peer ID         Peer =
hook     =20
>>  ----------      ---------       ---------    -------         =
---------     =20
>>  link0           em1             ether        00000003        lower   =
      =20
>>=20
>> =3D=3D=3D
>>=20
>> We're not done with the bridge yet. Because we foresee the =
possibility that it might be nice to be able to communicate with the =
jail that we're going to later hook into this bridge=85 we should hook =
the physical adapter's "upper" hook into the bridge.
>>=20
>> If you don't do this, you won't be able to (for example) ping a jail =
from the host where the host has only the PHY and the jail has only a =
(yet uncreated) eiface. Regardless of the fact that the bridge uses the =
PHY and the jail uses the bridge, to communicate with an IP that is =
configured on the base host, you must hook the upper.
>>=20
>> dteske@scu0a.jbsd.vicor.com ~ $ sudo ngctl connect em1: em1:lower =
upper link1
>>=20
>> If you want to use the alias I set up earlier (of "em1bridge") that =
works too (just don't forget the colon at the end of the alias):
>>=20
>> dteske@scu0a.jbsd.vicor.com ~ $ sudo ngctl connect em1: em1bridge: =
upper link1
>>=20
>> Here's the results:
>>=20
>> dteske@scu0a.jbsd.vicor.com ~ $ sudo ngctl ls -l
>> There are 5 total nodes:
>>  Name: em0             Type: ether           ID: 00000002   Num =
hooks: 0
>>=20
>>  Name: em1             Type: ether           ID: 00000003   Num =
hooks: 2
>>  Local hook      Peer name       Peer type    Peer ID         Peer =
hook     =20
>>  ----------      ---------       ---------    -------         =
---------     =20
>>  upper           em1bridge       bridge       0000002a        link1   =
      =20
>>  lower           em1bridge       bridge       0000002a        link0   =
      =20
>>=20
>>  Name: ngctl1874       Type: socket          ID: 00000030   Num =
hooks: 0
>>=20
>>  Name: em1bridge       Type: bridge          ID: 0000002a   Num =
hooks: 2
>>  Local hook      Peer name       Peer type    Peer ID         Peer =
hook     =20
>>  ----------      ---------       ---------    -------         =
---------     =20
>>  link1           em1             ether        00000003        upper   =
      =20
>>  link0           em1             ether        00000003        lower   =
      =20
>>=20
>>  Name: msk0            Type: ether           ID: 00000001   Num =
hooks: 0
>>=20
>>=20
>> NOTE: Some of the Peer ID's have changed, because I wanted to test =
that the alias could be used; I used "sudo ngctl shutdown em1bridge:" =
and re-executed up to the point where I connect the em1:upper into the =
bridge=85 except this time using the alias of "em1bridge" instead of =
"em1:lower" (indeed, you can use them interchangeably).
>>=20
>> =3D=3D=3D
>>=20
>> Ok=85 We've now done the hard part=85 which was to create and =
configure a bridge that is usable by any new nodes we connect to it and =
also (if you hooked the upper portion of em1 back into its own lower =
which is acting as the bridge) the base machine can communicate with any =
of the forth-coming jails (if on the same subnet at least).
>>=20
>> There's an easy step that shouldn't be skipped though=85
>>=20
>> Before you can truly use this bridge with any other interfaces=85
>>=20
>> dteske@scu0a.jbsd.vicor.com ~ $ sudo ifconfig em1 up
>> dteske@scu0a.jbsd.vicor.com ~ $ sudo ngctl msg em1: setpromisc 1
>> dteske@scu0a.jbsd.vicor.com ~ $ sudo ngctl msg em1: setautosrc 0
>>=20
>> A bridge cannot send packets out if the interface is down.
>> A bridge cannot work properly without promiscuous mode.
>> A bridge cannot send out packets for different addresses unless you =
turn off "setautosrc"
>>=20
>> =3D=3D=3D
>>=20
>> Let's create our first virtual NIC and connect it to the bridge.
>>=20
>> dteske@scu0a.jbsd.vicor.com ~ $ sudo ngctl mkpeer em1bridge: eiface =
link2 ether
>>=20
>> This command did two things. It created a new "eiface" node (see =
ng_eiface(4)), and connected it to the bridge.
>>=20
>> Let's have a look:
>>=20
>> dteske@scu0a.jbsd.vicor.com ~ $ sudo ngctl ls -l
>> There are 6 total nodes:
>>  Name: em0             Type: ether           ID: 00000002   Num =
hooks: 0
>>=20
>>  Name: em1             Type: ether           ID: 00000003   Num =
hooks: 2
>>  Local hook      Peer name       Peer type    Peer ID         Peer =
hook     =20
>>  ----------      ---------       ---------    -------         =
---------     =20
>>  upper           em1bridge       bridge       0000002a        link1   =
      =20
>>  lower           em1bridge       bridge       0000002a        link0   =
      =20
>>=20
>>  Name: ngeth0          Type: eiface          ID: 00000035   Num =
hooks: 1
>>  Local hook      Peer name       Peer type    Peer ID         Peer =
hook     =20
>>  ----------      ---------       ---------    -------         =
---------     =20
>>  ether           em1bridge       bridge       0000002a        link2   =
      =20
>>=20
>>  Name: ngctl2800       Type: socket          ID: 00000036   Num =
hooks: 0
>>=20
>>  Name: em1bridge       Type: bridge          ID: 0000002a   Num =
hooks: 3
>>  Local hook      Peer name       Peer type    Peer ID         Peer =
hook     =20
>>  ----------      ---------       ---------    -------         =
---------     =20
>>  link2           ngeth0          eiface       00000035        ether   =
      =20
>>  link1           em1             ether        00000003        upper   =
      =20
>>  link0           em1             ether        00000003        lower   =
      =20
>>=20
>>  Name: msk0            Type: ether           ID: 00000001   Num =
hooks: 0
>>=20
>> The list of hooks for our bridge (em1bridge) is growing, and now we =
see a new node (ngeth0) with one hook into that bridge.
>>=20
>> =3D=3D=3D
>>=20
>> ASIDE: If you wanted to script this=85 here's how you can test for an =
unused link:
>>=20
>> Right now, we have link0, link1, and link2 for the bridge. If a link =
exists for a bridge, the following command will return some info about =
the link and return success (whereas if the link does not exist, the =
command will return an error and exit with error-status):
>>=20
>> dteske@scu0a.jbsd.vicor.com ~ $ sudo ngctl msg em1bridge: getstats 0
>> Rec'd response "getstats" (4) from "[2a]:":
>> Args:
>> {}
>> dteske@scu0a.jbsd.vicor.com ~ $ sudo ngctl msg em1bridge: getstats 1
>> Rec'd response "getstats" (4) from "[2a]:":
>> Args:
>> {}
>> dteske@scu0a.jbsd.vicor.com ~ $ sudo ngctl msg em1bridge: getstats 2
>> Rec'd response "getstats" (4) from "[2a]:":
>> Args:
>> {}
>> dteske@scu0a.jbsd.vicor.com ~ $ sudo ngctl msg em1bridge: getstats 3
>> ngctl: send msg: Socket is not connected
>> dteske@scu0a.jbsd.vicor.com ~ $ sudo ngctl msg em1bridge: getstats 4
>> ngctl: send msg: Socket is not connected
>> dteske@scu0a.jbsd.vicor.com ~ $ sudo ngctl msg em1bridge: getstats 5
>> ngctl: send msg: Socket is not connected
>>=20
>> As you can see from the above output=85 we get errors for link3, =
link4, and link5, because they don't exist. Naturally, testing $? exit =
status after each of these commands would show how this can be scripted =
(HINT: throw stdout/stderr to /dev/null and test $?).
>>=20
>> =3D=3D=3D
>>=20
>> At this point=85 you say "ifconfig":
>>=20
>> dteske@oos0a.lbxrich.vicor.com ~ $ ifconfig
>> msk0: flags=3D8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
>>        =
options=3Dc011a<TXCSUM,VLAN_MTU,VLAN_HWTAGGING,TSO4,VLAN_HWTSO,LINKSTATE>
>>        ether xx:xx:xx:xx:xx:xx
>>        media: Ethernet autoselect
>> em0: flags=3D8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 =
mtu 1500
>>        =
options=3D209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC=
>
>>        ether xx:xx:xx:xx:xx:xx
>>        inet xx.xx.xx.xx netmask 0xffffff80 broadcast xx.xx.xx.xx
>>        media: Ethernet autoselect (1000baseT <full-duplex>)
>>        status: active
>> em1: flags=3D8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> =
metric 0 mtu 1500
>>        =
options=3D209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC=
>
>>        ether xx:xx:xx:xx:xx:xx
>>        media: Ethernet autoselect
>>        status: no carrier
>> ipfw0: flags=3D8801<UP,SIMPLEX,MULTICAST> metric 0 mtu 65536
>> lo0: flags=3D8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
>>        options=3D3<RXCSUM,TXCSUM>
>>        inet 127.0.0.1 netmask 0xff000000=20
>> ngeth0: flags=3D8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
>>        ether 00:00:00:00:00:00
>>=20
>> =3D=3D=3D
>>=20
>> Ok, there are two problems with the network interface.
>>=20
>> 1. It has a NULL MAC address (00:00:00:00:00:00). Good luck =
communicating on the Internet (remember, we disabled setautosrc -- we =
intend to make up a MAC address that is unique).
>>=20
>> 2. The name leaves something to be desired (if we're going to use =
this with a vimage jail, it would be nice if the interface had the jail =
name in it, so that when you do an "ngctl ls -l" or an "ngctl dot" =85 =
you're going to see the jail name so it becomes clear which jails are =
hooked to which PHY's through which bridges).
>>=20
>> =3D=3D=3D
>>=20
>> Let's tackle the easier one first=85 let's rename this new interface.
>>=20
>> You and I already know that this interface that we want to rename is =
"ngeth0"=85 but you can actually extract the name from the link in the =
bridge.
>>=20
>> dteske@scu0a.jbsd.vicor.com ~ $ sudo ngctl show -n em1bridge:link2
>>  Name: ngeth0          Type: eiface          ID: 00000035   Num =
hooks: 1
>>=20
>>=20
>> First, we rename it in netgraph (this does not affect the output of =
ifconfig -- and again, we do this to make "ngctl ls -l" and "ngctl dot" =
more palatable):
>>=20
>> dteske@scu0a.jbsd.vicor.com ~ $ sudo ngctl name em1bridge:link2 =
ng0_myjail
>> dteske@scu0a.jbsd.vicor.com ~ $ sudo ngctl ls -l  =20
>> There are 6 total nodes:
>>  Name: em0             Type: ether           ID: 00000002   Num =
hooks: 0
>>=20
>>  Name: em1             Type: ether           ID: 00000003   Num =
hooks: 2
>>  Local hook      Peer name       Peer type    Peer ID         Peer =
hook     =20
>>  ----------      ---------       ---------    -------         =
---------     =20
>>  upper           em1bridge       bridge       0000002a        link1   =
      =20
>>  lower           em1bridge       bridge       0000002a        link0   =
      =20
>>=20
>>  Name: ngctl2843       Type: socket          ID: 00000046   Num =
hooks: 0
>>=20
>>  Name: ng0_myjail      Type: eiface          ID: 00000035   Num =
hooks: 1
>>  Local hook      Peer name       Peer type    Peer ID         Peer =
hook     =20
>>  ----------      ---------       ---------    -------         =
---------     =20
>>  ether           em1bridge       bridge       0000002a        link2   =
      =20
>>=20
>>  Name: em1bridge       Type: bridge          ID: 0000002a   Num =
hooks: 3
>>  Local hook      Peer name       Peer type    Peer ID         Peer =
hook     =20
>>  ----------      ---------       ---------    -------         =
---------     =20
>>  link2           ng0_myjail      eiface       00000035        ether   =
      =20
>>  link1           em1             ether        00000003        upper   =
      =20
>>  link0           em1             ether        00000003        lower   =
      =20
>>=20
>>  Name: msk0            Type: ether           ID: 00000001   Num =
hooks: 0
>>=20
>>=20
>> Looking good. However, ifconfig hasn't changed=85
>>=20
>> dteske@scu0a.jbsd.vicor.com ~ $ ifconfig
>> ...
>> ngeth0: flags=3D8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
>> ether 00:00:00:00:00:00
>>=20
>> We want to rename the interface with ifconfig for a different reason.
>>=20
>> We renamed the interface with netgraph earlier so that netgraph =
outputs would be nice and easy to digest.
>>=20
>> This time, we rename with ifconfig so that we can layer jails onto =
the same rootdir.
>>=20
>> The naming convention (which is the same naming convention I use for =
renaming on the netgraph side) is:
>>=20
>> ng#_name
>>=20
>> The # always starts at zero for each jail where "name" is the name of =
the jail.
>>=20
>> Again=85 I use this scheme so that I can layer jails onto the same =
root-dir; /etc/rc.conf is then populated with things like:
>>=20
>> ifconfig_ng0_myjail=3D...
>> ifconfig_ng0_myrouter=3D...
>> ifconfig_ng1_myrouter=3D...
>> ifconfig_ng0_anotherjail=3D...
>>=20
>> So that when you say "service netif start" inside the vnet jail=85 it =
applies the right settings.
>>=20
>> So=85 we rename with ifconfig:
>>=20
>> dteske@scu0a.jbsd.vicor.com ~ $ sudo ifconfig ngeth0 name ng0_myjail
>> dteske@scu0a.jbsd.vicor.com ~ $ ifconfig
>> ...
>> ng0_myjail: flags=3D8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu =
1500
>> ether 00:00:00:00:00:00
>>=20
>> =3D=3D=3D
>>=20
>> We're almost ready to shove this interface into a jail (which we =
haven't created yet).
>>=20
>> But=85 we come back to that NULL MAC address.
>>=20
>> NOTE: Forming your own MAC address, or even coming up with your own =
formula should not be taken lightly.
>>=20
>> Here's a formula I use (which is based on several RFC's for MAC =
address formation):
>>=20
>> NOTE: In this context, ${_bridge} is em1 and $LINKNUM is 2
>>=20
>>                                # Set the MAC address of the new =
interface
>>                                # using a sensible algorithm to =
prevent
>>                                # conflicts on the network.
>>                                #
>>                                # MAC  LP:LL:LB:BB:BB:BB
>>                                # P    2, 6, A, or E but usually 2
>>                                # NOTE: Indicates "privately =
administered" MAC
>>                                # L    ng_bridge(4) link number =
(1-65535)
>>                                # B    Same as bridged interface
>>                                #
>>                                _bridge_ether=3D$( ifconfig ${_bridge} =
ether |
>>                                        awk '/ether/{print $2}' )
>>                                =
_ether_devid=3D"${_bridge_ether#??:??:?}"
>>                                n=3D$LINKNUM
>>                                _quad=3D$(($n & 15))
>>                                case "${_quad}" in
>>                                10) _quad=3Da;; 11) _quad=3Db;; 12) =
_quad=3Dc;;
>>                                13) _quad=3Dd;; 14) _quad=3De;; 15) =
_quad=3Df;;
>>                                esac
>>                                =
_ether_devid=3D":${_quad}${_ether_devid}"
>>                                n=3D$(($n >> 4))
>>                                _quad=3D$(($n & 15))
>>                                case "${_quad}" in
>>                                10) _quad=3Da;; 11) _quad=3Db;; 12) =
_quad=3Dc;;
>>                                13) _quad=3Dd;; 14) _quad=3De;; 15) =
_quad=3Df;;
>>                                esac
>>                                _ether_devid=3D"${_quad}${_ether_devid}"=

>>                                n=3D$(($n >> 4))
>>                                _quad=3D$(($n & 15))
>>                                case "${_quad}" in
>>                                10) _quad=3Da;; 11) _quad=3Db;; 12) =
_quad=3Dc;;
>>                                13) _quad=3Dd;; 14) _quad=3De;; 15) =
_quad=3Df;;
>>                                esac
>>                                =
_ether_devid=3D"2:${_quad}${_ether_devid}"
>>                                n=3D$(($n >> 4))
>>                                _quad=3D$(($n & 15))
>>                                case "${_quad}" in
>>                                10) _quad=3Da;; 11) _quad=3Db;; 12) =
_quad=3Dc;;
>>                                13) _quad=3Dd;; 14) _quad=3De;; 15) =
_quad=3Df;;
>>                                esac
>>                                _ether_devid=3D"${_quad}${_ether_devid}"=

>>                                n=3D$(($n >> 4))
>>=20
>> After which=85 ${_ether_devid}  holds a properly formed MAC address =
that can (in every case I've tested) "get out".
>>=20
>> Here's what I do to set it:
>>=20
>> ifconfig ng0_myjail ether "${_ether_devid}"
>>=20
>> Here's an example of how the MAC address was translated from the =
physical adapter to the ng_eiface(4) interface:
>>=20
>> dteske@scu0a.jbsd.vicor.com ~ $ ifconfig em1; ifconfig ng0_myjail
>> em1: flags=3D8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> =
metric 0 mtu 1500
>> =
options=3D209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC=
>
>> ether 00:0e:0c:ab:1b:76
>> media: Ethernet autoselect
>> status: no carrier
>> ng0_myjail: flags=3D8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu =
1500
>> ether 02:00:2c:ab:1b:76
>>=20
>> =3D=3D=3D
>>=20
>> OK=85 we're now ready to shove that interface into a vimage jail.
>>=20
>> But=85
>>=20
>> First we need a vimage jail. (this is not a tutorial on how to =
create, manage, build, or do anything else with jails, vimage-jails, or =
vps-jails *other* than give it a netgraph based interface)
>>=20
>> I'm going to use my existing base machine as a fake jail (by pointing =
my jail's rootdir at "/").
>>=20
>> NOTE: Certain sysctl's have to be set appropriately before you fire =
up the jail to make this vimage jail able to do "more" on the net.
>>=20
>> dteske@scu0a.jbsd.vicor.com ~ $ sudo sysctl =
security.jail.set_hostname_allowed=3D1 security.jail.sysvipc_allowed=3D1 =
security.jail.socket_unixiproute_only=3D1
>> security.jail.set_hostname_allowed: 1 -> 1
>> security.jail.sysvipc_allowed: 1 -> 1
>> security.jail.socket_unixiproute_only: 0 -> 1
>>=20
>> NOTE: Unless you intend to reboot to restore the defaults later=85 =
you might want to take down those previous values for restoration =
*after* we fire up the "vimage" jail.
>>=20
>> dteske@scu0a.jbsd.vicor.com ~ $ sudo jail -i -c vnet name=3Dmyjail =
host.hostname=3Dmyjail path=3D/ persist
>> 1
>> dteske@scu0a.jbsd.vicor.com ~ $ jls
>>   JID  IP Address      Hostname                      Path
>>     1  -               myjail                        /
>>=20
>> OK=85 we have a running jail (with the vnet property, making it a =
"vimage" jail -- which can accept network interfaces).
>>=20
>> =3D=3D=3D
>>=20
>> Right now our jail has no network interfaces (well, it has an =
unconfigured lo0).
>>=20
>> dteske@scu0a.jbsd.vicor.com ~ $ sudo jexec myjail ifconfig
>> lo0: flags=3D8008<LOOPBACK,MULTICAST> metric 0 mtu 16384
>> options=3D3<RXCSUM,TXCSUM>
>>=20
>> So let's pass the netgraph created interface into the jail=85
>>=20
>> dteske@scu0a.jbsd.vicor.com ~ $ sudo ifconfig ng0_myjail vnet 1
>> dteske@scu0a.jbsd.vicor.com ~ $ sudo jexec myjail ifconfig
>> lo0: flags=3D8008<LOOPBACK,MULTICAST> metric 0 mtu 16384
>> options=3D3<RXCSUM,TXCSUM>
>> ng0_myjail: flags=3D8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu =
1500
>> ether 02:00:2c:ab:1b:76
>>=20
>> Sweet!
>>=20
>> =3D=3D=3D
>>=20
>> Almost there=85
>>=20
>> Let's go into /etc/rc.conf, give it an IP, and start the network=85
>>=20
>> dteske@scu0a.jbsd.vicor.com ~ $ sudo sysrc ifconfig_ng0_myjail=3D"inet =
192.168.1.1 netmask 255.255.255.0"
>> /etc/rc.conf: ifconfig_ng0_myjail:  -> inet 192.168.1.1 netmask =
255.255.255.0
>> dteske@scu0a.jbsd.vicor.com ~ $ grep ng0 /etc/rc.conf
>> ifconfig_ng0_myjail=3D"inet 192.168.1.1 netmask 255.255.255.0"
>> dteske@scu0a.jbsd.vicor.com ~ $ sudo jexec myjail service netif start
>> Starting Network: lo0 ng0_myjail.
>> lo0: flags=3D8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
>> options=3D3<RXCSUM,TXCSUM>
>> inet 127.0.0.1 netmask 0xff000000=20
>> ng0_myjail: flags=3D8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> =
metric 0 mtu 1500
>> ether 02:00:2c:ab:1b:76
>> inet 192.168.1.1 netmask 0xffffff00 broadcast 192.168.1.255
>>=20
>> Now we're cookin' with gasoline!
>>=20
>> =3D=3D=3D
>>=20
>> Optionally go configure your base machine with an IP and have fun.
>=20
> A quick conclusion=85
>=20
> Because we've built this all on top of netgraph=85 we can =85 graph =
it.
>=20
> dteske@scu0a.jbsd.vicor.com ~ $ sudo ngctl dot | dot -Tsvg -o =
netgraph-scu0a.svg
>=20
> I then uploaded the file to the web and here it is:
>=20
> http://druidbsd.sourceforge.net/download/netgraph-scu0a.svg
>=20
> You should compare this directly to the output of "ngctl ls -l":
>=20
> dteske@scu0a.jbsd.vicor.com ~ $ sudo ngctl ls -l
> There are 6 total nodes:
>  Name: em0             Type: ether           ID: 00000002   Num hooks: =
0
>=20
>  Name: em1             Type: ether           ID: 00000003   Num hooks: =
2
>  Local hook      Peer name       Peer type    Peer ID         Peer =
hook     =20
>  ----------      ---------       ---------    -------         =
---------     =20
>  upper           em1bridge       bridge       0000002a        link1    =
     =20
>  lower           em1bridge       bridge       0000002a        link0    =
     =20
>=20
>  Name: ng0_myjail      Type: eiface          ID: 00000035   Num hooks: =
1
>  Local hook      Peer name       Peer type    Peer ID         Peer =
hook     =20
>  ----------      ---------       ---------    -------         =
---------     =20
>  ether           em1bridge       bridge       0000002a        link2    =
     =20
>=20
>  Name: em1bridge       Type: bridge          ID: 0000002a   Num hooks: =
3
>  Local hook      Peer name       Peer type    Peer ID         Peer =
hook     =20
>  ----------      ---------       ---------    -------         =
---------     =20
>  link2           ng0_myjail      eiface       00000035        ether    =
     =20
>  link1           em1             ether        00000003        upper    =
     =20
>  link0           em1             ether        00000003        lower    =
     =20
>=20
>  Name: ngctl8676       Type: socket          ID: 00000049   Num hooks: =
0
>=20
>  Name: msk0            Type: ether           ID: 00000001   Num hooks: =
0
>=20
> You'll notice that when you graph the layout with "ngctl dot", the =
nodes are rendered as boxes displaying their "Peer Name" up top, their =
"Peer Type" in the lower-left, and their "Peer ID" in the bottom-right.
>=20
> The edges from one node to another contains two octagons. These are =
the "Local hook" and "Peer hook".
> --=20
> Devin
>=20
> _____________
> The information contained in this message is proprietary and/or =
confidential. If you are not the intended recipient, please: (i) delete =
the message and all copies; (ii) do not disclose, distribute or use the =
message in any manner; and (iii) notify the sender immediately. In =
addition, please be aware that any message addressed to our domain is =
subject to archiving and review by persons other than the intended =
recipient. Thank you.

Hello Devin,

If you live in the same city I will invite you for a couple of beer (I =
have to pay of course!) - I live in the Netherlands then let me know -=20=


I followed your tutorial with the expected result: I can ping the em1 =
interface but I still have the same problems of before regarding =
external to internal networks communication.

Please note that on my original host (no jail) the default gateway is =
192.168.1.254, that's important for what I am going to do.

I added a default route like :

route add default -interface ng0_myjail=20

and then I try to :

jexec myjail ping 8.8.8.8

I analyzed the wireshark capture and I can see that an ARP request for =
the 192.168.254 (with the MAC address of our virtual NIC as source, as =
expected) go out my freebsd env - which is on a virtualbox - and it gets =
the answer but when I read the ARP table of the jail I can see it as =
incomplete. It seems that packet going out but the answer is not =
received and it is confirmed when I try to sniff with tcpdump on my =
FREEBSD, I can't see any ARP request going in both from my physical and =
virtual NIC, the same if I try :

ping 192.168.1.254

Then I can see ping reply coming from wireshark but not from tcpdump on =
FreeBDS.

Wireshark is attached on the host machine on the physical interface =
where VB is attached in Bridged Mode, my original physical interface on =
FreeBSD took the IP address from DHCP without problem then the problem =
is related to the jail.

do I try do accomplish a task which is not possible with JAIL or =
something is wrong in my configuration or worst, in my brain :P ?

Thanks in advance,
Pietro.









Want to link to this message? Use this URL: <http://docs.FreeBSD.org/cgi/mid.cgi?93DE0F1F-FF8C-4A48-BE9B-70C0F0B84AE8>