From owner-freebsd-questions@FreeBSD.ORG Sat May 18 23:29:56 2013 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 545ED2C3 for ; Sat, 18 May 2013 23:29:56 +0000 (UTC) (envelope-from Devin.Teske@fisglobal.com) Received: from mx1.fisglobal.com (mx1.fisglobal.com [199.200.24.190]) by mx1.freebsd.org (Postfix) with ESMTP id 1B4AD956 for ; Sat, 18 May 2013 23:29:55 +0000 (UTC) Received: from smtp.fisglobal.com ([10.132.206.16]) by ltcfislmsgpa01.fnfis.com (8.14.5/8.14.5) with ESMTP id r4INTkDB024757 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NOT); Sat, 18 May 2013 18:29:46 -0500 Received: from LTCFISWMSGMB26.FNFIS.com ([10.132.99.18]) by LTCFISWMSGHT05.FNFIS.com ([10.132.206.16]) with mapi id 14.02.0309.002; Sat, 18 May 2013 18:29:45 -0500 From: "Teske, Devin" To: Joe Subject: Re: netgraph network setup for jail(8) vnet jails. Thread-Topic: netgraph network setup for jail(8) vnet jails. Thread-Index: AQHOU93CBuO69ayRZkOyIHdwpxatO5kL6zyA Date: Sat, 18 May 2013 23:29:45 +0000 Message-ID: <13CA24D6AB415D428143D44749F57D7201F5076A@ltcfiswmsgmb26> References: <5197A06A.9080401@a1poweruser.com> In-Reply-To: <5197A06A.9080401@a1poweruser.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.132.253.126] Content-Type: text/plain; charset="us-ascii" Content-ID: Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:5.10.8626, 1.0.431, 0.0.0000 definitions=2013-05-18_08:2013-05-17,2013-05-18,1970-01-01 signatures=0 Cc: "" X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: Devin Teske List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 18 May 2013 23:29:56 -0000 Sorry for top-post, but just wanted to add a quick note: The output of "ngctl dot" would be very helpful to others in debugging your= setup. --=20 Devin On May 18, 2013, at 8:38 AM, Joe wrote: > Hello list >=20 > I cant get to the internet using this netgraph setup script. > I sure would appreciate giving this console log a look over for > errors. My netgraph knowledge level is not sufficient to see what is > wrong. The goal is to run this script to setup and break down a netgraph > network for a single vnet jail at a time. rl0 is the real nic interface > device name of the nic facing the internet. This box is on my lan and > the gateway box does NAT for all lan boxes. The host running this script = can ping the internet ok. >=20 > Thank you very much for your help. >=20 >=20 >=20 >=20 >=20 > The host's kernel has modules with vimage & ipfw compiled in. >=20 > From the host > # /root >ifconfig > rl0: flags=3D8843 metric 0 mtu > options=3D2008 > ether 00:0c:6e:09:8b:74 > inet 10.0.10.5 netmask 0xfffffff8 broadcast 10.0.10.7 > nd6 options=3D29 > media: Ethernet autoselect (100baseTX ) > status: active > plip0: flags=3D8810 metric 0 mtu 1500 > nd6 options=3D29 > ipfw0: flags=3D8801 metric 0 mtu 65536 > nd6 options=3D29 > lo0: flags=3D8049 metric 0 mtu 16384 > options=3D600003 > inet6 ::1 prefixlen 128 > inet6 fe80::1%lo0 prefixlen 64 scopeid 0x8 > inet 127.0.0.1 netmask 0xff000000 > nd6 options=3D21 >=20 > The jails config file > # /root >cat /usr/local/etc/vnet/vdir4 > vdir4 { > host.hostname =3D "vdir4"; > path =3D "/usr/jails/vdir4"; > mount.fstab =3D "/usr/local/etc/fstab/vdir4"; > vnet; > persist; > } >=20 > The netgraph script > # /root >cat /usr/local/bin/vnet.ng.test > #!/bin/sh > # snip comments for displaying here > # This script is based on this /usr/share/examples/netgraph/virtual.lan >=20 > # Give the name of ethernet interface. > ETHER_INTF=3D"rl0" >=20 > # List the names of virtual nodes and their IP addresses. Use ':' > # character to separate node name from node IP address and netmask. >=20 > #TARGET_TOPOLOGY=3D"c1|10.0.2.20/24 c2|10.0.2.21/24 c3|10.0.2.22/24" > TARGET_TOPOLOGY=3D"vdir4|10.0.2.20/24" >=20 > # MAC manufacturer prefix. This can be modified according to needs. > MAC_PREFIX=3D"00:1d:92" >=20 > # Temporary file is important for proper execution of script. > TEMP_FILE=3D"/var/tmp/virtual.lan.tmp" >=20 > virtual_lan_start() { >=20 > # Load netgraph KLD's as necessary. >=20 > for KLD in ng_ether ng_bridge ng_eiface; do > if ! kldstat -v | grep -qw ${KLD}; then > echo -n "Loading ${KLD}.ko... " > kldload ${KLD} || exit 1 > echo "done" > fi > done >=20 > # Reset all interfaces and jails. If temporary file can not be found > # script assumes that there is no previous configuration. >=20 > if [ ! -e ${TEMP_FILE} ]; then > echo "No previous configuration(${TEMP_FILE}) found to clean-up." > else > echo -n "Cleaning previous configuration..." > virtual_lan_stop > echo "done" > fi >=20 > # Create temporary file for usage. This file includes generated > # interface names and jail names. All bridges, interfaces and jails > # are written to file while created. In clean-up process written > # objects are cleaned (i.e. removed) from system. >=20 > if [ -e ${TEMP_FILE} ]; then > touch ${TEMP_FILE} > fi >=20 > echo -n "Verifying ethernet interface existence..." > # Verify ethernet interface exist. > if ! ngctl info ${ETHER_INTF}: >/dev/null 2>&1; then > echo "Error: interface ${ETHER_INTF} does not exist" > exit 1 > fi >=20 > ifconfig ${ETHER_INTF} up || exit 1 > echo "done" >=20 > # Get current number of bridge interfaces in the system. This number > # is used to create a name for new bridge. > BRIDGE_COUNT=3D`ngctl l | grep bridge | wc -l | sed -e "s/ //g"` > BRIDGE_NAME=3D"bridge${BRIDGE_COUNT}" >=20 > # Create new ng_bridge(4) node and attach it to the ethernet interface. > # Connect ng_ether:lower hook to bridge:link0 when creating bridge and > # connect ng_ether:upper hook to bridge:link1 after bridge name is set. >=20 > echo "Creating bridge interface: ${BRIDGE_NAME}..." > ngctl mkpeer ${ETHER_INTF}: bridge lower link0 || exit 1 > ngctl name ${ETHER_INTF}:lower ${BRIDGE_NAME} || exit 1 > ngctl connect ${ETHER_INTF}: ${BRIDGE_NAME}: upper link1 || exit 1 > echo "Bridge ${BRIDGE_NAME} is created and ${ETHER_INTF} is connected." >=20 > # In the above code block two hooks are connected to bridge interface, > # therefore LINKNUM is set to 2 indicating total number of connected > # hooks on the bridge interface. > LINKNUM=3D2 >=20 > # Write name of the bridge to temp file. Clean-up procedure will use > # this name to shutdown bridge interface. > echo "bridge ${BRIDGE_NAME}" > ${TEMP_FILE} >=20 >=20 > # Attach vnet jail. > for NODE in ${TARGET_TOPOLOGY}; do >=20 > # Virtual nodes are defined in TARGET_TOPOLOGY variable. They > # have the form of 'nodeName|IPaddr'. Below two lines split > # node definition to get node name and node IP. >=20 > NODE_NAME=3D`echo ${NODE} | awk -F"|" '{print $1}'` > NODE_IP=3D`echo ${NODE} | awk -F"|" '{print $2}'` >=20 > # Create virtual node (jail) with given name > echo -n "Creating virtual node (jail) ${NODE_NAME}..." >=20 > jail -f "/usr/local/etc/vnet/${NODE_NAME}" -c ${NODE_NAME} > if [ $? -ne 0 ]; then > echo "Error: /usr/sbin/jail failed to start jail ${NODE_NAME}." > virtual_lan_stop > exit 2 > fi > echo "done" >=20 > # Write name of the jail to temp file. Clean-up procedure will > # use this name to remove jail. >=20 > echo "node ${NODE_NAME}" >> ${TEMP_FILE} >=20 > # Create a ng_eiface object for virtual node. ng_eiface > # object has a hook that can be connected to one of bridge > # links. After creating interface get its automatically > # generated name for further usage. >=20 > echo "Creating eiface interface for virtual node ${NODE_NAME}." > ngctl mkpeer eiface ether ether > EIFACE=3D`ngctl l | grep ngeth | tail -n 1| awk '{print $2}'` > echo "Interface ${EIFACE} is created." >=20 > # Write name of the interface to temp file. Clean-up procedure > # will use this name to shutdown interface. >=20 > echo "interface ${EIFACE}" >> ${TEMP_FILE} >=20 > # Move virtual interface to virtual node. Note that Interface > # name will not be changed at the end of this movement. Moved > # interface can be seen at the output of ifconfig command in > # jail: 'jexec jailname ifconfig' >=20 > echo "Moving ${EIFACE} to ${NODE_NAME}" > ifconfig ${EIFACE} vnet ${NODE_NAME} >=20 > # Make lo0 interface localhost. > jexec ${NODE_NAME} ifconfig lo0 localhost >=20 > # Generate a random mac address for virtual interface. First > # three octets can be changed by user. Last three octets are > # generated randomly. > M4=3D`od -An -N2 -i /dev/random | sed -e 's/ //g' | \ > awk '{ print $1 % 256 }'` > M5=3D`od -An -N2 -i /dev/random | sed -e 's/ //g' | \ > awk '{ print $1 % 256 }'` > M6=3D`od -An -N2 -i /dev/random | sed -e 's/ //g' | \ > awk '{ print $1 % 256 }'` >=20 > MAC=3D`printf ${MAC_PREFIX}:%02x:%02x:%02x ${M4} ${M5} ${M6}` >=20 > # Set the link address (mac address) of virtual interface in > # virtual node to randomly generated MAC. > echo "Setting MAC address of ${EIFACE} to '${MAC}'" > jexec ${NODE_NAME} ifconfig ${EIFACE} link $MAC >=20 > # Either IPv4 or IPv6 can be used in this script. Ifconfig > # IP setting syntax differs slightly for two IP versions. > # For version 4 'inet' keyword is used whereas for version 6 > # 'inet6' is used. Below line tries to decide which IP version > # is given and sets IPVER to 'inet' or 'inet6'. >=20 > IPVER=3D`echo ${NODE_IP} | awk -F"." '{ split($4,last,"/"); \ > if( NF=3D=3D4 && $1>0 && $1<256 && $2<256 && $3<256 && \ > last[1]<256) print "inet"; else print "inet6"}'` >=20 > # Set IP address of virtual interface in virtual node. > echo "Setting IP address of ${EIFACE} to '${NODE_IP}'" > jexec ${NODE_NAME} ifconfig ${EIFACE} ${IPVER} ${NODE_IP} >=20 > # Connect virtual interface to bridge interface. Syntax is : > # ngctl connect INTERFACE: BRIDGE: INTERFACE_HOOK EMPTY_LINK. > # Interface has one hook named 'ether' and below line connects > # ether hook to bridge's first unconnected link. >=20 > echo -n "Connecting ${EIFACE}:ether to ${BRIDGE_NAME}:link${LINKNUM}..." >=20 > ngctl connect ${EIFACE}: ${BRIDGE_NAME}: ether link${LINKNUM} \ > || exit 1 > echo "done" >=20 > # Now, bridge has one more connected link thus link count is > # incremented. > LINKNUM=3D`expr ${LINKNUM} + 1` > done > echo "Virtual LAN established successfully!" >=20 > } >=20 >=20 > # Stop routine. > virtual_lan_stop() { >=20 > if [ ! -e ${TEMP_FILE} ]; then > echo "Nothing to stop! ${TEMP_FILE}: temp file not found" > else > echo -n "Shutdown bridge interface.." > OBJECTS=3D`cat ${TEMP_FILE} | grep bridge | awk '{print $2}'` > for BRIDGE in ${OBJECTS}; do > ngctl shutdown ${BRIDGE}: >/dev/null 2>&1 > done > echo "done" >=20 > echo -n "Shutdown all eiface interfaces..." > OBJECTS=3D`cat ${TEMP_FILE} | grep interface | awk '{print $2}'` > for INTERFACE in ${OBJECTS}; do > ngctl shutdown ${INTERFACE}: >/dev/null 2>&1 > done > echo "done" >=20 > echo -n "Removing all jails..." > OBJECTS=3D`cat ${TEMP_FILE} | grep node | awk '{print $2}'` > for NODE in ${OBJECTS}; do > jail -f "/usr/local/etc/vnet/${NODE}" -r ${NODE} > done > echo "done" >=20 > echo "Removing tempfile ${TEMP_FILE}" > rm ${TEMP_FILE} > fi > echo "Virtual LAN objects removed successfully!" >=20 > } >=20 >=20 > # Main entry point. >=20 > case $# in > 1) > case $1 in > start) > echo -n "Creating default target topology:" > echo " ${TARGET_TOPOLOGY}" > virtual_lan_start > ;; > stop) >=20 > if [ ! -e ${TEMP_FILE} ]; then > echo -n "Noting to stop! ${TEMP_FILE}:" > echo " temp file not found" > else > virtual_lan_stop > fi > ;; > help) > virtual_lan_usage > exit 1 > ;; > *) > virtual_lan_usage > exit 1 >=20 > esac > ;; > 2) > case $1 in > start) > TARGET_TOPOLOGY=3D$2 > echo -n "Creating target topology:" > echo "${TARGET_TOPOLOGY}" > virtual_lan_start > ;; > *) > virtual_lan_usage > exit 1 > esac > ;; >=20 > *) > virtual_lan_usage > exit 1 > esac >=20 >=20 > # /root >vnet.ng.test start > Creating default target topology: vdir4|10.0.2.20/24 > Loading ng_ether.ko... done > Loading ng_bridge.ko... done > Loading ng_eiface.ko... done > No previous configuration(/var/tmp/virtual.lan.tmp) found to clean-up. > Verifying ethernet interface existence...done > Creating bridge interface: bridge0... > Bridge bridge0 is created and rl0 is connected. > Creating virtual node (jail) vdir4...vdir4: created > done > Creating eiface interface for virtual node vdir4. > Interface ngeth0 is created. > Moving ngeth0 to vdir4 > Setting MAC address of ngeth0 to '00:1d:92:df:92:8e' > Setting IP address of ngeth0 to '10.0.2.20/24' > Connecting ngeth0:ether to bridge0:link2...done > Virtual LAN established successfully! >=20 >=20 > # /root >ngctl ls -l > There are 5 total nodes: > Name: rl0 Type: ether ID: 00000001 Num hooks: 2 > Local hook Peer name Peer type Peer ID Peer > hook > ---------- --------- --------- ------- > --------- > upper bridge0 bridge 00000006 link1 >=20 > lower bridge0 bridge 00000006 link0 >=20 > Name: ipfw0 Type: ether ID: 00000002 Num hooks: 0 > Name: bridge0 Type: bridge ID: 00000006 Num hooks: 3 > Local hook Peer name Peer type Peer ID Peer > hook > ---------- --------- --------- ------- > --------- > link2 ngeth0 eiface 0000000a ether >=20 > link1 rl0 ether 00000001 upper >=20 > link0 rl0 ether 00000001 lower >=20 > Name: ngeth0 Type: eiface ID: 0000000a Num hooks: 1 > Local hook Peer name Peer type Peer ID Peer > hook > ---------- --------- --------- ------- > --------- > ether bridge0 bridge 00000006 link2 >=20 > Name: ngctl1513 Type: socket ID: 0000000d Num hooks: 0 >=20 >=20 > # /root >jexec vdir4 tcsh > vdir4 / >ping -c 1 8.8.178.135 > PING 8.8.178.135 (8.8.178.135): 56 data bytes > ping: sendto: No route to host >=20 > --- 8.8.178.135 ping statistics --- > 1 packets transmitted, 0 packets received, 100.0% packet loss > vdir4 / >exit > exit >=20 >=20 > # /root >vnet.ng.test stop > Shutdown bridge interface..done > Shutdown all eiface interfaces...done > Removing all jails...vdir4: removed > done > Removing tempfile /var/tmp/virtual.lan.tmp > Virtual LAN objects removed successfully! >=20 > # /root >jls > JID IP Address Hostname Path >=20 > # /root >ngctl ls -l > There are 3 total nodes: > Name: ngctl1540 Type: socket ID: 00000010 Num hooks: 0 > Name: rl0 Type: ether ID: 00000001 Num hooks: 0 > Name: ipfw0 Type: ether ID: 00000002 Num hooks: 0 >=20 >=20 >=20 >=20 > _______________________________________________ > freebsd-questions@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-questions > To unsubscribe, send any mail to "freebsd-questions-unsubscribe@freebsd.o= rg" _____________ The information contained in this message is proprietary and/or confidentia= l. If you are not the intended recipient, please: (i) delete the message an= d all copies; (ii) do not disclose, distribute or use the message in any ma= nner; and (iii) notify the sender immediately. In addition, please be aware= that any message addressed to our domain is subject to archiving and revie= w by persons other than the intended recipient. Thank you.