From owner-freebsd-questions@FreeBSD.ORG Sat May 18 15:38:26 2013 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 89285C50 for ; Sat, 18 May 2013 15:38:26 +0000 (UTC) (envelope-from fbsd8@a1poweruser.com) Received: from mail-03.name-services.com (mail-03.name-services.com [69.64.155.195]) by mx1.freebsd.org (Postfix) with ESMTP id 7929C765 for ; Sat, 18 May 2013 15:38:26 +0000 (UTC) Received: from [10.0.10.1] ([173.88.196.224]) by mail-03.name-services.com with Microsoft SMTPSVC(6.0.3790.4675); Sat, 18 May 2013 08:38:21 -0700 Message-ID: <5197A06A.9080401@a1poweruser.com> Date: Sat, 18 May 2013 11:38:18 -0400 From: Joe User-Agent: Thunderbird 2.0.0.17 (Windows/20080914) MIME-Version: 1.0 To: freebsd-questions@freebsd.org Subject: netgraph network setup for jail(8) vnet jails. Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-OriginalArrivalTime: 18 May 2013 15:38:21.0612 (UTC) FILETIME=[BA23F6C0:01CE53DD] X-Sender: fbsd8@a1poweruser.com X-Authenticated-Sender: fbsd8@a1poweruser.com X-EchoSenderHash: [fbsd8]-[a1poweruser*com] X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 18 May 2013 15:38:26 -0000 Hello list I cant get to the internet using this netgraph setup script. I sure would appreciate giving this console log a look over for errors. My netgraph knowledge level is not sufficient to see what is wrong. The goal is to run this script to setup and break down a netgraph network for a single vnet jail at a time. rl0 is the real nic interface device name of the nic facing the internet. This box is on my lan and the gateway box does NAT for all lan boxes. The host running this script can ping the internet ok. Thank you very much for your help. The host's kernel has modules with vimage & ipfw compiled in. From the host # /root >ifconfig rl0: flags=8843 metric 0 mtu options=2008 ether 00:0c:6e:09:8b:74 inet 10.0.10.5 netmask 0xfffffff8 broadcast 10.0.10.7 nd6 options=29 media: Ethernet autoselect (100baseTX ) status: active plip0: flags=8810 metric 0 mtu 1500 nd6 options=29 ipfw0: flags=8801 metric 0 mtu 65536 nd6 options=29 lo0: flags=8049 metric 0 mtu 16384 options=600003 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x8 inet 127.0.0.1 netmask 0xff000000 nd6 options=21 The jails config file # /root >cat /usr/local/etc/vnet/vdir4 vdir4 { host.hostname = "vdir4"; path = "/usr/jails/vdir4"; mount.fstab = "/usr/local/etc/fstab/vdir4"; vnet; persist; } The netgraph script # /root >cat /usr/local/bin/vnet.ng.test #!/bin/sh # snip comments for displaying here # This script is based on this /usr/share/examples/netgraph/virtual.lan # Give the name of ethernet interface. ETHER_INTF="rl0" # List the names of virtual nodes and their IP addresses. Use ':' # character to separate node name from node IP address and netmask. #TARGET_TOPOLOGY="c1|10.0.2.20/24 c2|10.0.2.21/24 c3|10.0.2.22/24" TARGET_TOPOLOGY="vdir4|10.0.2.20/24" # MAC manufacturer prefix. This can be modified according to needs. MAC_PREFIX="00:1d:92" # Temporary file is important for proper execution of script. TEMP_FILE="/var/tmp/virtual.lan.tmp" virtual_lan_start() { # Load netgraph KLD's as necessary. for KLD in ng_ether ng_bridge ng_eiface; do if ! kldstat -v | grep -qw ${KLD}; then echo -n "Loading ${KLD}.ko... " kldload ${KLD} || exit 1 echo "done" fi done # Reset all interfaces and jails. If temporary file can not be found # script assumes that there is no previous configuration. if [ ! -e ${TEMP_FILE} ]; then echo "No previous configuration(${TEMP_FILE}) found to clean-up." else echo -n "Cleaning previous configuration..." virtual_lan_stop echo "done" fi # Create temporary file for usage. This file includes generated # interface names and jail names. All bridges, interfaces and jails # are written to file while created. In clean-up process written # objects are cleaned (i.e. removed) from system. if [ -e ${TEMP_FILE} ]; then touch ${TEMP_FILE} fi echo -n "Verifying ethernet interface existence..." # Verify ethernet interface exist. if ! ngctl info ${ETHER_INTF}: >/dev/null 2>&1; then echo "Error: interface ${ETHER_INTF} does not exist" exit 1 fi ifconfig ${ETHER_INTF} up || exit 1 echo "done" # Get current number of bridge interfaces in the system. This number # is used to create a name for new bridge. BRIDGE_COUNT=`ngctl l | grep bridge | wc -l | sed -e "s/ //g"` BRIDGE_NAME="bridge${BRIDGE_COUNT}" # Create new ng_bridge(4) node and attach it to the ethernet interface. # Connect ng_ether:lower hook to bridge:link0 when creating bridge and # connect ng_ether:upper hook to bridge:link1 after bridge name is set. echo "Creating bridge interface: ${BRIDGE_NAME}..." ngctl mkpeer ${ETHER_INTF}: bridge lower link0 || exit 1 ngctl name ${ETHER_INTF}:lower ${BRIDGE_NAME} || exit 1 ngctl connect ${ETHER_INTF}: ${BRIDGE_NAME}: upper link1 || exit 1 echo "Bridge ${BRIDGE_NAME} is created and ${ETHER_INTF} is connected." # In the above code block two hooks are connected to bridge interface, # therefore LINKNUM is set to 2 indicating total number of connected # hooks on the bridge interface. LINKNUM=2 # Write name of the bridge to temp file. Clean-up procedure will use # this name to shutdown bridge interface. echo "bridge ${BRIDGE_NAME}" > ${TEMP_FILE} # Attach vnet jail. for NODE in ${TARGET_TOPOLOGY}; do # Virtual nodes are defined in TARGET_TOPOLOGY variable. They # have the form of 'nodeName|IPaddr'. Below two lines split # node definition to get node name and node IP. NODE_NAME=`echo ${NODE} | awk -F"|" '{print $1}'` NODE_IP=`echo ${NODE} | awk -F"|" '{print $2}'` # Create virtual node (jail) with given name echo -n "Creating virtual node (jail) ${NODE_NAME}..." jail -f "/usr/local/etc/vnet/${NODE_NAME}" -c ${NODE_NAME} if [ $? -ne 0 ]; then echo "Error: /usr/sbin/jail failed to start jail ${NODE_NAME}." virtual_lan_stop exit 2 fi echo "done" # Write name of the jail to temp file. Clean-up procedure will # use this name to remove jail. echo "node ${NODE_NAME}" >> ${TEMP_FILE} # Create a ng_eiface object for virtual node. ng_eiface # object has a hook that can be connected to one of bridge # links. After creating interface get its automatically # generated name for further usage. echo "Creating eiface interface for virtual node ${NODE_NAME}." ngctl mkpeer eiface ether ether EIFACE=`ngctl l | grep ngeth | tail -n 1| awk '{print $2}'` echo "Interface ${EIFACE} is created." # Write name of the interface to temp file. Clean-up procedure # will use this name to shutdown interface. echo "interface ${EIFACE}" >> ${TEMP_FILE} # Move virtual interface to virtual node. Note that Interface # name will not be changed at the end of this movement. Moved # interface can be seen at the output of ifconfig command in # jail: 'jexec jailname ifconfig' echo "Moving ${EIFACE} to ${NODE_NAME}" ifconfig ${EIFACE} vnet ${NODE_NAME} # Make lo0 interface localhost. jexec ${NODE_NAME} ifconfig lo0 localhost # Generate a random mac address for virtual interface. First # three octets can be changed by user. Last three octets are # generated randomly. M4=`od -An -N2 -i /dev/random | sed -e 's/ //g' | \ awk '{ print $1 % 256 }'` M5=`od -An -N2 -i /dev/random | sed -e 's/ //g' | \ awk '{ print $1 % 256 }'` M6=`od -An -N2 -i /dev/random | sed -e 's/ //g' | \ awk '{ print $1 % 256 }'` MAC=`printf ${MAC_PREFIX}:%02x:%02x:%02x ${M4} ${M5} ${M6}` # Set the link address (mac address) of virtual interface in # virtual node to randomly generated MAC. echo "Setting MAC address of ${EIFACE} to '${MAC}'" jexec ${NODE_NAME} ifconfig ${EIFACE} link $MAC # Either IPv4 or IPv6 can be used in this script. Ifconfig # IP setting syntax differs slightly for two IP versions. # For version 4 'inet' keyword is used whereas for version 6 # 'inet6' is used. Below line tries to decide which IP version # is given and sets IPVER to 'inet' or 'inet6'. IPVER=`echo ${NODE_IP} | awk -F"." '{ split($4,last,"/"); \ if( NF==4 && $1>0 && $1<256 && $2<256 && $3<256 && \ last[1]<256) print "inet"; else print "inet6"}'` # Set IP address of virtual interface in virtual node. echo "Setting IP address of ${EIFACE} to '${NODE_IP}'" jexec ${NODE_NAME} ifconfig ${EIFACE} ${IPVER} ${NODE_IP} # Connect virtual interface to bridge interface. Syntax is : # ngctl connect INTERFACE: BRIDGE: INTERFACE_HOOK EMPTY_LINK. # Interface has one hook named 'ether' and below line connects # ether hook to bridge's first unconnected link. echo -n "Connecting ${EIFACE}:ether to ${BRIDGE_NAME}:link${LINKNUM}..." ngctl connect ${EIFACE}: ${BRIDGE_NAME}: ether link${LINKNUM} \ || exit 1 echo "done" # Now, bridge has one more connected link thus link count is # incremented. LINKNUM=`expr ${LINKNUM} + 1` done echo "Virtual LAN established successfully!" } # Stop routine. virtual_lan_stop() { if [ ! -e ${TEMP_FILE} ]; then echo "Nothing to stop! ${TEMP_FILE}: temp file not found" else echo -n "Shutdown bridge interface.." OBJECTS=`cat ${TEMP_FILE} | grep bridge | awk '{print $2}'` for BRIDGE in ${OBJECTS}; do ngctl shutdown ${BRIDGE}: >/dev/null 2>&1 done echo "done" echo -n "Shutdown all eiface interfaces..." OBJECTS=`cat ${TEMP_FILE} | grep interface | awk '{print $2}'` for INTERFACE in ${OBJECTS}; do ngctl shutdown ${INTERFACE}: >/dev/null 2>&1 done echo "done" echo -n "Removing all jails..." OBJECTS=`cat ${TEMP_FILE} | grep node | awk '{print $2}'` for NODE in ${OBJECTS}; do jail -f "/usr/local/etc/vnet/${NODE}" -r ${NODE} done echo "done" echo "Removing tempfile ${TEMP_FILE}" rm ${TEMP_FILE} fi echo "Virtual LAN objects removed successfully!" } # Main entry point. case $# in 1) case $1 in start) echo -n "Creating default target topology:" echo " ${TARGET_TOPOLOGY}" virtual_lan_start ;; stop) if [ ! -e ${TEMP_FILE} ]; then echo -n "Noting to stop! ${TEMP_FILE}:" echo " temp file not found" else virtual_lan_stop fi ;; help) virtual_lan_usage exit 1 ;; *) virtual_lan_usage exit 1 esac ;; 2) case $1 in start) TARGET_TOPOLOGY=$2 echo -n "Creating target topology:" echo "${TARGET_TOPOLOGY}" virtual_lan_start ;; *) virtual_lan_usage exit 1 esac ;; *) virtual_lan_usage exit 1 esac # /root >vnet.ng.test start Creating default target topology: vdir4|10.0.2.20/24 Loading ng_ether.ko... done Loading ng_bridge.ko... done Loading ng_eiface.ko... done No previous configuration(/var/tmp/virtual.lan.tmp) found to clean-up. Verifying ethernet interface existence...done Creating bridge interface: bridge0... Bridge bridge0 is created and rl0 is connected. Creating virtual node (jail) vdir4...vdir4: created done Creating eiface interface for virtual node vdir4. Interface ngeth0 is created. Moving ngeth0 to vdir4 Setting MAC address of ngeth0 to '00:1d:92:df:92:8e' Setting IP address of ngeth0 to '10.0.2.20/24' Connecting ngeth0:ether to bridge0:link2...done Virtual LAN established successfully! # /root >ngctl ls -l There are 5 total nodes: Name: rl0 Type: ether ID: 00000001 Num hooks: 2 Local hook Peer name Peer type Peer ID Peer hook ---------- --------- --------- ------- --------- upper bridge0 bridge 00000006 link1 lower bridge0 bridge 00000006 link0 Name: ipfw0 Type: ether ID: 00000002 Num hooks: 0 Name: bridge0 Type: bridge ID: 00000006 Num hooks: 3 Local hook Peer name Peer type Peer ID Peer hook ---------- --------- --------- ------- --------- link2 ngeth0 eiface 0000000a ether link1 rl0 ether 00000001 upper link0 rl0 ether 00000001 lower Name: ngeth0 Type: eiface ID: 0000000a Num hooks: 1 Local hook Peer name Peer type Peer ID Peer hook ---------- --------- --------- ------- --------- ether bridge0 bridge 00000006 link2 Name: ngctl1513 Type: socket ID: 0000000d Num hooks: 0 # /root >jexec vdir4 tcsh vdir4 / >ping -c 1 8.8.178.135 PING 8.8.178.135 (8.8.178.135): 56 data bytes ping: sendto: No route to host --- 8.8.178.135 ping statistics --- 1 packets transmitted, 0 packets received, 100.0% packet loss vdir4 / >exit exit # /root >vnet.ng.test stop Shutdown bridge interface..done Shutdown all eiface interfaces...done Removing all jails...vdir4: removed done Removing tempfile /var/tmp/virtual.lan.tmp Virtual LAN objects removed successfully! # /root >jls JID IP Address Hostname Path # /root >ngctl ls -l There are 3 total nodes: Name: ngctl1540 Type: socket ID: 00000010 Num hooks: 0 Name: rl0 Type: ether ID: 00000001 Num hooks: 0 Name: ipfw0 Type: ether ID: 00000002 Num hooks: 0