Date: Mon, 13 Oct 2014 09:23:55 -0600 From: Alan Somers <asomers@freebsd.org> To: Harald Schmalzbauer <h.schmalzbauer@omnilan.de> Cc: "freebsd-net@freebsd.org" <freebsd-net@freebsd.org>, "Alexander V. Chernikov" <melifaro@freebsd.org>, FreeBSD <freebsd-stable@freebsd.org> Subject: Re: Deleting IPv4 iface-routes from extra FIBs Message-ID: <CAOtMX2gS%2BN5c1McqAAPG_q4fWey-06DW08i5mGuM5cxmksF2Og@mail.gmail.com> In-Reply-To: <543B9873.3040605@omnilan.de> References: <53569ABA.60007@omnilan.de> <CA%2BP_MZH_iScuJ4S=xiKocnEwTzT1eRJPNpJKbboZDfG3B=TBzA@mail.gmail.com> <535771F3.4070007@freebsd.org> <543B8ED5.6040206@omnilan.de> <543B9075.2000102@FreeBSD.org> <543B9873.3040605@omnilan.de>
next in thread | previous in thread | raw e-mail | index | archive | help
On Mon, Oct 13, 2014 at 3:16 AM, Harald Schmalzbauer
<h.schmalzbauer@omnilan.de> wrote:
> Bez=C3=BCglich Alexander V. Chernikov's Nachricht vom 13.10.2014 10:42
> (localtime):
>> On 13.10.2014 12:35, Harald Schmalzbauer wrote:
>>> Bez=C3=BCglich Julian Elischer's Nachricht vom 23.04.2014 09:55
>>> (localtime):
> ...
>>>> yes, we made two behaviours.
>>>> Add interface routes to all active FIBS or only add them to the first
>>>> fib and let the user populate other fibs as needed.
>>>> It appears you want the second behaviour, so I suggest you use that
>>>> option and set up all your routes manually.
>>> Hello,
>>>
>>> last time I had the iface-route problem, I just reverted r248895 (for
>>> 9.3). There was inconsitent behaviour with v6 iface routes and
>>> net.add_addr_allfibs=3D0.
>>> Now I checked with 10.1 ans it seems net.add_addr_allfibs=3D0 doesn't w=
ork
>>> any more:
>>> netstat -f inet -nr
>>> Routing tables
>>>
>>> Internet:
>>> Destination Gateway Flags Netif Expire
>>> default 172.21.32.1 UGS egn
>>> 127.0.0.1 link#2 UH lo0
>>> 172.21.32.0/19 link#1 U egn
>>> 172.21.35.1 link#1 UHS lo0
>>>
>>> netstat -F 1 -f inet -nr
>>> Routing tables (fib: 1)
>>>
>>> Internet:
>>> Destination Gateway Flags Netif Expire
>>> 127.0.0.1 link#2 UH lo0
>>> 172.21.32.0/19 link#1 U egn
>>>
>>> 'sysctl net.add_addr_allfibs'
>>> net.add_addr_allfibs: 0
>> Are you sure net.add_addr_allfibs was applied before interface address
>> added?
>
> Sorry, I messed it up. Forgot that on my production systems (where I
> tested), / is read-only with /etc as union-mount.
> Adding net.add_addr_allfibs=3D0 to the correct sysctl.conf made the inet
> routing table stay empty.
>
> But unfortunately not the inet6 routing table :-(
> So I still need to delete iface routes for my jail setups, hence need to
> revert r248895.
What do your ipv6 routing tables look like when the sysctl is set
correctly and 248895 is in place?
>
> <Unrelated to the actual problem:
> Strage thing is that 'rcorder' shows nothing iface related before
> mountcritlocal, where I resource /etc/rc.d, so the
> 'net.add_addr_allfibs' in my union-mounted sysctl.conf should work!?!
> But that's my homemade problem ;-) />
>
> For those having similar problems, here's how I currently solve my jail
> setups:
>
> jail.conf:
>
> jail {
> allow.set_hostname;
> ...
> exec.fib =3D 1;
> exec.prestart =3D "/bin/sh /.JAIL$name/etc/rc.jails_fibprepare -f
> 1 -i inop";
> interface =3D inop;
> ...
>
> =E2=80=93=E2=80=93=E2=80=93
> rc.jails_fibprepare :
>
> #!/bin/sh
> # format FIB for JAIL usage (remove all but own interface routes)
> # Does only work if on FreeBSD-9.2 if r248895 was reverted, since
> deleting iface routes is prohibited by default.
> # TODO: extend jail (8) and jail.conf for routing parameters and delete
> this ugly hack!
> # TODO: Do it the other way, not deleleting, but adding if "sysctl
> net.add_addr_allfibs=3D0".
> # Last edited: 20140605.0
>
>
> _help(){
> echo "Usage: rc.jails_fibprepare -f FIBNUM -i IFACENAME [-4
> defaultrouterIPv4] [-6 defaultrouterIPv6] [-h]"
> if [ "X$1" !=3D "X" ]; then
> if [ "$1" =3D "-h" ]; then
> echo "Prepare routing tabel of specified FIB for jail usage."
> echo "This removes all iface routes not belonging to own interface"
> echo "and sets default route(s) if specified or automatically, if"
> echo "iface used is the same where fib 0 has set the default gatewa=
y."
> echo " -f: FIBNUM is the number of the fib whose routing
> table will be altered."
> echo " -i: IFACENAME is the name of the interface we have
> our IP on."
> echo " -4: IP (v4) of the defaultrouter."
> echo " -6: IP (v6) of the defaultrouter."
> echo " -h: This help"
> echo
> else
> echo "ERROR:"
> echo " $1"
> echo
> exit 1
> fi
> else
> echo "Type \"rc.jails_fibprepare -h\" for more help."
> exit 1
> fi
> exit 0
> }
>
> _find_unwanted_destinations(){
> # first, generate complete destination lists (separate for v4+v6)
> dest4list=3D`setfib ${fibnum} netstat -f inet -nr | grep -E
> '^[[:print:]]+(%[[:alnum:].]+|[[:digit:]])[[:blank:]]+U[[:print:]]+$' |
> cut -s -d ' ' -f 1`
> dest6list=3D`setfib ${fibnum} netstat -f inet6 -nr | grep -E
> '^[[:print:]]+(%[[:alnum:].]+|[[:digit:]])[[:blank:]]+U[[:print:]]+$' |
> cut -s -d ' ' -f 1`
> # Create lists with wanted destinations (separate for v4+v6)
> for ifn in ${ifnames}; do
> link=3D`setfib ${fibnum} netstat -I ${ifn} | sed -n -E
> 's/^[[:print:]]+<[lL](ink#[[:digit:]]{1,2})>[[:print:]]+$/l\1/p'`
> dest4wanted=3D"`setfib ${fibnum} netstat -f inet -nr | grep -E
> '^[^[:blank:]]+[[:blank:]]+'"${link}"'[[:blank:]]+.*$' | cut -s -d ' '
> -f 1` ${dest4wanted:-}"
> dest6wanted=3D"`setfib ${fibnum} netstat -f inet6 -nr | grep -E
> '^[^[:blank:]]+[[:blank:]]+'"${link}"'[[:blank:]]+.*$' | cut -s -d ' '
> -f 1` ${dest6wanted:-}"
> done
> # remove wanted destinations from v4 list
> for dest in ${dest4wanted}; do
> dest4list=3D"`echo ${dest4list} | sed -E 's,'"${dest}"' *,,'`"
> done
> # remove wanted destinations from v6 list
> for dest in ${dest6wanted}; do
> dest6list=3D"`echo ${dest6list} | sed -E 's,'"${dest}"' *,,'`"
> done
> }
>
> _clean_fib(){
> _find_unwanted_destinations || return 1
> # extract default gateway IPv4 if it's on one of our interfaces and
> none is set already
> for ifn in ${ifnames}; do
> if [ "X${dv4gw}" =3D "X" ]; then
> dv4gw=3D"`netstat -f inet -nr | sed -n -E
> 's/^default[[:print:]]+[[:blank:]]([^[:blank:]]+[.:][^[:blank:]]+)[[:prin=
t:]]+[^[:blank:]]+[[:blank:]]+'"${ifn}"'$/\1/p'`"
> fi
> done
> # extract default gateway IPv6 if it's on one of our interfaces and
> none is set already
> for ifn in ${ifnames}; do
> if [ "X${dv6gw}" =3D "X" ]; then
> dv6gw=3D"`netstat -f inet6 -nr | sed -n -E
> 's/^default[[:print:]]+[[:blank:]]([^[:blank:]]+[.:][^[:blank:]]+)[[:prin=
t:]]+[^[:blank:]]+[[:blank:]]+'"${ifn}"'$/\1/p'`"
> fi
> done
> # remove v4 destinations
> for dest in ${dest4list}; do
> route -q delete -net -inet ${dest} -fib ${fibnum} || return 1
> done
> # remove v6 destinations
> for dest in ${dest6list}; do
> route -q delete -net -inet6 ${dest} -fib ${fibnum} || return 1
> done
> # Set v4 defaultrouter
> if [ "X${dv4gw}" !=3D "X" ]; then
> route -q add -net -inet default ${dv4gw} -fib ${fibnum} || return 1
> fi
> # Set v6 defaultrouter
> if [ "X${dv6gw}" !=3D "X" ]; then
> route -q add -net -inet6 default ${dv6gw} -fib ${fibnum} || return 1
> fi
> }
>
> if [ $# -gt 8 ]; then
> _help "Too many arguments!"
> else
> if [ $# -lt 4 ]; then
> _help "At least \"-f FIBUM\" and \"-i IFACENAME\" is required!"
> else
> if ! expr $# % 2 >/dev/null; then
> while [ $# -gt 0 ]; do
> case "$1" in
> -f) if ! setfib ${2} true; then
> _help "FIBNUM too high!"
> else
> fibnum=3D$2
> fi
> ;;
> -i) if ! ifconfig ${2} >/dev/null 2>&1; then
> _help "No such interface: \"$2\""
> else
> ifnames=3D"$2 ${ifnames:-}"
> fi
> ;;
> -4) dv4gw=3D"$2";;
> -6) dv6gw=3D"$2";;
> -h|*) _help "$1"
> esac
> shift 2
> done
> _clean_fib && exit 0
> else
> _help "Wrong number of arguments ($#), only even numbers can be
> valid!"
> fi
> fi
> fi
> exit 1
>
> =E2=80=93=E2=80=93=E2=80=93
> r248895-revert patch against 10.1:
>
> --- src/sys/net/if.c 2014-10-06 12:56:27.000000000 +0200
> +++ src/sys/net/if.c 2014-10-13 10:47:51.000000000 +0200
> @@ -1371,8 +1371,7 @@
> return (0);
>
> err =3D rtrequest_fib(RTM_DELETE, rt_key(rt), rt->rt_gateway,
> - rt_mask(rt),
> - rt->rt_flags|RTF_RNH_LOCKED|RTF_PINNED,
> + rt_mask(rt), rt->rt_flags|RTF_RNH_LOCKED,
> (struct rtentry **) NULL, rt->rt_fibnum);
> if (err) {
> log(LOG_WARNING, "if_rtdel: error %d\n", err);
> --- src/sys/net/route.c 2014-10-06 12:56:27.000000000 +0200
> +++ src/sys/net/route.c 2014-10-13 10:47:51.000000000 +0200
> @@ -1210,14 +1210,6 @@
> error =3D 0;
> }
> #endif
> - if ((flags & RTF_PINNED) =3D=3D 0) {
> - /* Check if target route can be deleted */
> - rt =3D (struct rtentry *)rnh->rnh_lookup(dst,
> - netmask, rnh);
> - if ((rt !=3D NULL) && (rt->rt_flags & RTF_PINNED))
> - senderr(EADDRINUSE);
> - }
> -
> /*
> * Remove the item from the tree and return it.
> * Complain if it is not there and do no more processing.
> @@ -1521,7 +1513,6 @@
> int didwork =3D 0;
> int a_failure =3D 0;
> static struct sockaddr_dl null_sdl =3D {sizeof(null_sdl), AF_LINK};
> - struct radix_node_head *rnh;
>
> if (flags & RTF_HOST) {
> dst =3D ifa->ifa_dstaddr;
> @@ -1580,6 +1571,7 @@
> */
> for ( fibnum =3D startfib; fibnum <=3D endfib; fibnum++) {
> if (cmd =3D=3D RTM_DELETE) {
> + struct radix_node_head *rnh;
> struct radix_node *rn;
> /*
> * Look up an rtentry that is in the routing tree and
> @@ -1626,8 +1618,7 @@
> */
> bzero((caddr_t)&info, sizeof(info));
> info.rti_ifa =3D ifa;
> - info.rti_flags =3D flags |
> - (ifa->ifa_flags & ~IFA_RTSELF) | RTF_PINNED;
> + info.rti_flags =3D flags | (ifa->ifa_flags & ~IFA_RTSELF);
> info.rti_info[RTAX_DST] =3D dst;
> /*
> * doing this for compatibility reasons
> @@ -1639,33 +1630,6 @@
> info.rti_info[RTAX_GATEWAY] =3D ifa->ifa_addr;
> info.rti_info[RTAX_NETMASK] =3D netmask;
> error =3D rtrequest1_fib(cmd, &info, &rt, fibnum);
> -
> - if ((error =3D=3D EEXIST) && (cmd =3D=3D RTM_ADD)) {
> - /*
> - * Interface route addition failed.
> - * Atomically delete current prefix generating
> - * RTM_DELETE message, and retry adding
> - * interface prefix.
> - */
> - rnh =3D rt_tables_get_rnh(fibnum, dst->sa_family);
> - RADIX_NODE_HEAD_LOCK(rnh);
> -
> - /* Delete old prefix */
> - info.rti_ifa =3D NULL;
> - info.rti_flags =3D RTF_RNH_LOCKED;
> -
> - error =3D rtrequest1_fib(RTM_DELETE, &info, NULL, fibnum);
> - if (error =3D=3D 0) {
> - info.rti_ifa =3D ifa;
> - info.rti_flags =3D flags | RTF_RNH_LOCKED |
> - (ifa->ifa_flags & ~IFA_RTSELF) | RTF_PINNED;
> - error =3D rtrequest1_fib(cmd, &info, &rt, fibnum);
> - }
> -
> - RADIX_NODE_HEAD_UNLOCK(rnh);
> - }
> -
> -
> if (error =3D=3D 0 && rt !=3D NULL) {
> /*
> * notify any listening routing agents of the change
> --- src/sys/net/route.h 2014-10-06 12:56:27.000000000 +0200
> +++ src/sys/net/route.h 2014-10-13 10:43:59.000000000 +0200
> @@ -148,7 +148,7 @@
> /* 0x20000 unused, was RTF_WASCLONED */
> #define RTF_PROTO3 0x40000 /* protocol specific routing flag *=
/
> /* 0x80000 unused */
> -#define RTF_PINNED 0x100000 /* route is immutable */
> +#define RTF_PINNED 0x100000 /* future use (route is immutable,
> startintg with r248895) */
> #define RTF_LOCAL 0x200000 /* route represents a local address=
*/
> #define RTF_BROADCAST 0x400000 /* route represents a bcast
> address */
> #define RTF_MULTICAST 0x800000 /* route represents a mcast
> address */
>
>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAOtMX2gS%2BN5c1McqAAPG_q4fWey-06DW08i5mGuM5cxmksF2Og>