Date: Thu, 10 Jun 2010 16:55:41 GMT From: Haven Hash <haven.hash@isilon.com> To: freebsd-gnats-submit@FreeBSD.org Subject: kern/147767: route -n change operation with INVARIANTS kernel causes mutex radix node head not owned ASSERTION failure Message-ID: <201006101655.o5AGtfsq026200@www.freebsd.org> Resent-Message-ID: <201006101700.o5AH0Dsk065153@freefall.freebsd.org>
next in thread | raw e-mail | index | archive | help
>Number: 147767
>Category: kern
>Synopsis: route -n change operation with INVARIANTS kernel causes mutex radix node head not owned ASSERTION failure
>Confidential: no
>Severity: non-critical
>Priority: low
>Responsible: freebsd-bugs
>State: open
>Quarter:
>Keywords:
>Date-Required:
>Class: sw-bug
>Submitter-Id: current-users
>Arrival-Date: Thu Jun 10 17:00:13 UTC 2010
>Closed-Date:
>Last-Modified:
>Originator: Haven Hash
>Release: 7.3 Release
>Organization:
Isilon
>Environment:
FreeBSD 7.3-RELEASE FreeBSD 7.3-RELEASE #1: Thu Jun 10 01:24:18 UTC 2010 root@:/usr/obj/usr/src/sys/GENERIC i386
>Description:
When running certain route operations via the userspace route utility on kernels built with the following options:
INVARIANTS
INVARIANT_SUPPORT
a lock assertion causes a kernel panic, the panic requires these options because the assertion is only checked when INVARIANT support is enabled, however it seems likely that other consistency problems could occur on non-INVARIANT kernels.
>How-To-Repeat:
Compile and install a kernel with the following two lines included in the kernel configuration file:
option INVARIANTS
option INVARIANT_SUPPORT
If your machine does not have multiple external interfaces then you can create multiple vlan interfaces to substitute in for the following examples.
Add an IP address to an external interface (i.e. ifconfig em0 1.1.1.1 netmask 255.255.0.0).
Add another IP address in the same subnet to another external interface (i.e. ifconfig em1 1.1.1.2 netmask 255.255.0.0)
Attempt to use the route command to move the subnet route associated with the previously assigned IP addresses to go out the other interface (i.e. route -n change 1.1.0.0 -netmask 255.255.0.0 -ifp em1)
With INVARIANTS enabled a RNH (radix node head) lock assertion is hit.
A partial stack from one machine is as follows:
panic @ time 1276188167.945, thread 0x907f8000: mutex radix node head not owned
Panic occurred in module kernel loaded at 0x80200000:
Stack: --------------------------------------------------
kernel:_mtx_assert+0x72
kernel:rt_setgate+0xb9
kernel:arp_rtrequest+0x91
kernel:route_output+0x8d7
kernel:raw_usend+0x63
kernel:rts_send+0x27
kernel:sosend_generic+0x4f7
kernel:sosend+0x2e
>Fix:
It looks like rev: 189026 (a collection of merged revisions) added this assert and from the stack it looks like the calling of rt_setgate() from arp_rtrequest() does not take the necessary RNH lock being checked for. This code path is not the same in 8.x (nor have I been able to reproduce the issue on 8.x) as the arp code has been restructured in that line.
Attached patch which works for me, basically looks up the appropriate RNH from the global rt_tables and other available local variables, locks it before calling rt_setgate and unlocks it afterwards. Modified from similar code in route_output which does take this lock before calling rt_setgate().
Patch attached with submission follows:
Index: sys/netinet/if_ether.c
===================================================================
--- sys/netinet/if_ether.c (revision 206544)
+++ sys/netinet/if_ether.c (working copy)
@@ -154,6 +154,7 @@
static struct sockaddr_dl null_sdl = {sizeof(null_sdl), AF_LINK};
struct in_ifaddr *ia;
struct ifaddr *ifa;
+ struct radix_node_head *rnh;
RT_LOCK_ASSERT(rt);
@@ -177,8 +178,15 @@
/*
* Case 1: This route should come from a route to iface.
*/
+ rnh = rt_tables[rt->rt_fibnum][info->rti_info[RTAX_DST]->sa_family];
+ if (rnh == NULL)
+ break;
+ RT_UNLOCK(rt);
+ RADIX_NODE_HEAD_LOCK(rnh);
+ RT_LOCK(rt);
rt_setgate(rt, rt_key(rt),
(struct sockaddr *)&null_sdl);
+ RADIX_NODE_HEAD_UNLOCK(rnh);
gate = rt->rt_gateway;
SDL(gate)->sdl_type = rt->rt_ifp->if_type;
SDL(gate)->sdl_index = rt->rt_ifp->if_index;
>Release-Note:
>Audit-Trail:
>Unformatted:
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?201006101655.o5AGtfsq026200>