From owner-svn-src-user@FreeBSD.ORG Sat Dec 6 09:38:41 2008 Return-Path: Delivered-To: svn-src-user@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D57E7106567A; Sat, 6 Dec 2008 09:38:41 +0000 (UTC) (envelope-from kmacy@FreeBSD.org) Received: from svn.freebsd.org (svn.freebsd.org [IPv6:2001:4f8:fff6::2c]) by mx1.freebsd.org (Postfix) with ESMTP id C2AEC8FC0A; Sat, 6 Dec 2008 09:38:41 +0000 (UTC) (envelope-from kmacy@FreeBSD.org) Received: from svn.freebsd.org (localhost [127.0.0.1]) by svn.freebsd.org (8.14.3/8.14.3) with ESMTP id mB69cffO089506; Sat, 6 Dec 2008 09:38:41 GMT (envelope-from kmacy@svn.freebsd.org) Received: (from kmacy@localhost) by svn.freebsd.org (8.14.3/8.14.3/Submit) id mB69cfP3089495; Sat, 6 Dec 2008 09:38:41 GMT (envelope-from kmacy@svn.freebsd.org) Message-Id: <200812060938.mB69cfP3089495@svn.freebsd.org> From: Kip Macy Date: Sat, 6 Dec 2008 09:38:41 +0000 (UTC) To: src-committers@freebsd.org, svn-src-user@freebsd.org X-SVN-Group: user MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cc: Subject: svn commit: r185691 - in user/kmacy/HEAD_fast_multi_xmit/sys: amd64/conf contrib/pf/net kern net netinet6 X-BeenThere: svn-src-user@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "SVN commit messages for the experimental " user" src tree" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 06 Dec 2008 09:38:41 -0000 Author: kmacy Date: Sat Dec 6 09:38:41 2008 New Revision: 185691 URL: http://svn.freebsd.org/changeset/base/185691 Log: - convert radix node head lock from mutex to rwlock - make radix node head lock not recursive - fix LOR in rtexpunge - fix LOR in rtredirect Modified: user/kmacy/HEAD_fast_multi_xmit/sys/amd64/conf/PERFTEST user/kmacy/HEAD_fast_multi_xmit/sys/contrib/pf/net/pf_table.c user/kmacy/HEAD_fast_multi_xmit/sys/kern/subr_witness.c user/kmacy/HEAD_fast_multi_xmit/sys/kern/vfs_export.c user/kmacy/HEAD_fast_multi_xmit/sys/net/radix.c user/kmacy/HEAD_fast_multi_xmit/sys/net/radix.h user/kmacy/HEAD_fast_multi_xmit/sys/net/route.c user/kmacy/HEAD_fast_multi_xmit/sys/net/route.h user/kmacy/HEAD_fast_multi_xmit/sys/net/rtsock.c user/kmacy/HEAD_fast_multi_xmit/sys/netinet6/in6_rmx.c user/kmacy/HEAD_fast_multi_xmit/sys/netinet6/nd6_rtr.c Modified: user/kmacy/HEAD_fast_multi_xmit/sys/amd64/conf/PERFTEST ============================================================================== --- user/kmacy/HEAD_fast_multi_xmit/sys/amd64/conf/PERFTEST Sat Dec 6 09:37:54 2008 (r185690) +++ user/kmacy/HEAD_fast_multi_xmit/sys/amd64/conf/PERFTEST Sat Dec 6 09:38:41 2008 (r185691) @@ -279,3 +279,5 @@ device fwe # Ethernet over FireWire (n device fwip # IP over FireWire (RFC 2734,3146) device dcons # Dumb console driver device dcons_crom # Configuration ROM for dcons + +options ALT_BREAK_TO_DEBUGGER Modified: user/kmacy/HEAD_fast_multi_xmit/sys/contrib/pf/net/pf_table.c ============================================================================== --- user/kmacy/HEAD_fast_multi_xmit/sys/contrib/pf/net/pf_table.c Sat Dec 6 09:37:54 2008 (r185690) +++ user/kmacy/HEAD_fast_multi_xmit/sys/contrib/pf/net/pf_table.c Sat Dec 6 09:38:41 2008 (r185691) @@ -43,6 +43,8 @@ __FBSDID("$FreeBSD$"); #include #include #include +#include +#include #ifdef __FreeBSD__ #include #endif Modified: user/kmacy/HEAD_fast_multi_xmit/sys/kern/subr_witness.c ============================================================================== --- user/kmacy/HEAD_fast_multi_xmit/sys/kern/subr_witness.c Sat Dec 6 09:37:54 2008 (r185690) +++ user/kmacy/HEAD_fast_multi_xmit/sys/kern/subr_witness.c Sat Dec 6 09:38:41 2008 (r185691) @@ -505,7 +505,7 @@ static struct witness_order_list_entry o * Routing */ { "so_rcv", &lock_class_mtx_sleep }, - { "radix node head", &lock_class_mtx_sleep }, + { "radix node head", &lock_class_rw }, { "rtentry", &lock_class_mtx_sleep }, { "ifaddr", &lock_class_mtx_sleep }, { NULL, NULL }, Modified: user/kmacy/HEAD_fast_multi_xmit/sys/kern/vfs_export.c ============================================================================== --- user/kmacy/HEAD_fast_multi_xmit/sys/kern/vfs_export.c Sat Dec 6 09:37:54 2008 (r185690) +++ user/kmacy/HEAD_fast_multi_xmit/sys/kern/vfs_export.c Sat Dec 6 09:38:41 2008 (r185691) @@ -46,6 +46,7 @@ __FBSDID("$FreeBSD$"); #include #include #include +#include #include #include #include @@ -425,10 +426,10 @@ vfs_export_lookup(struct mount *mp, stru saddr = nam; rnh = nep->ne_rtable[saddr->sa_family]; if (rnh != NULL) { - RADIX_NODE_HEAD_LOCK(rnh); + RADIX_NODE_HEAD_RLOCK(rnh); np = (struct netcred *) (*rnh->rnh_matchaddr)(saddr, rnh); - RADIX_NODE_HEAD_UNLOCK(rnh); + RADIX_NODE_HEAD_RUNLOCK(rnh); if (np && np->netc_rnodes->rn_flags & RNF_ROOT) np = NULL; } Modified: user/kmacy/HEAD_fast_multi_xmit/sys/net/radix.c ============================================================================== --- user/kmacy/HEAD_fast_multi_xmit/sys/net/radix.c Sat Dec 6 09:37:54 2008 (r185690) +++ user/kmacy/HEAD_fast_multi_xmit/sys/net/radix.c Sat Dec 6 09:38:41 2008 (r185691) @@ -38,6 +38,7 @@ #ifdef _KERNEL #include #include +#include #include #include #include Modified: user/kmacy/HEAD_fast_multi_xmit/sys/net/radix.h ============================================================================== --- user/kmacy/HEAD_fast_multi_xmit/sys/net/radix.h Sat Dec 6 09:37:54 2008 (r185690) +++ user/kmacy/HEAD_fast_multi_xmit/sys/net/radix.h Sat Dec 6 09:38:41 2008 (r185691) @@ -36,6 +36,7 @@ #ifdef _KERNEL #include #include +#include #endif #ifdef MALLOC_DECLARE @@ -132,7 +133,7 @@ struct radix_node_head { struct radix_node rnh_nodes[3]; /* empty tree for common case */ int rnh_multipath; /* multipath capable ? */ #ifdef _KERNEL - struct mtx rnh_mtx; /* locks entire radix tree */ + struct rwlock rnh_lock; /* locks entire radix tree */ #endif }; @@ -146,11 +147,16 @@ struct radix_node_head { #define Free(p) free((caddr_t)p, M_RTABLE); #define RADIX_NODE_HEAD_LOCK_INIT(rnh) \ - mtx_init(&(rnh)->rnh_mtx, "radix node head", NULL, MTX_DEF | MTX_RECURSE) -#define RADIX_NODE_HEAD_LOCK(rnh) mtx_lock(&(rnh)->rnh_mtx) -#define RADIX_NODE_HEAD_UNLOCK(rnh) mtx_unlock(&(rnh)->rnh_mtx) -#define RADIX_NODE_HEAD_DESTROY(rnh) mtx_destroy(&(rnh)->rnh_mtx) -#define RADIX_NODE_HEAD_LOCK_ASSERT(rnh) mtx_assert(&(rnh)->rnh_mtx, MA_OWNED) + rw_init_flags(&(rnh)->rnh_lock, "radix node head", 0) +#define RADIX_NODE_HEAD_LOCK(rnh) rw_wlock(&(rnh)->rnh_lock) +#define RADIX_NODE_HEAD_UNLOCK(rnh) rw_wunlock(&(rnh)->rnh_lock) +#define RADIX_NODE_HEAD_RLOCK(rnh) rw_rlock(&(rnh)->rnh_lock) +#define RADIX_NODE_HEAD_RUNLOCK(rnh) rw_runlock(&(rnh)->rnh_lock) +#define RADIX_NODE_HEAD_LOCK_TRY_UPGRADE(rnh) rw_try_upgrade(&(rnh)->rnh_lock) + + +#define RADIX_NODE_HEAD_DESTROY(rnh) rw_destroy(&(rnh)->rnh_lock) +#define RADIX_NODE_HEAD_LOCK_ASSERT(rnh) rw_assert(&(rnh)->rnh_lock, RA_LOCKED) #endif /* _KERNEL */ void rn_init(void); Modified: user/kmacy/HEAD_fast_multi_xmit/sys/net/route.c ============================================================================== --- user/kmacy/HEAD_fast_multi_xmit/sys/net/route.c Sat Dec 6 09:37:54 2008 (r185690) +++ user/kmacy/HEAD_fast_multi_xmit/sys/net/route.c Sat Dec 6 09:38:41 2008 (r185691) @@ -45,6 +45,7 @@ #include #include #include +#include #include #include #include @@ -269,7 +270,8 @@ rtalloc1_fib(struct sockaddr *dst, int r struct rtentry *newrt; struct rt_addrinfo info; u_long nflags; - int err = 0, msgtype = RTM_MISS; + int needresolve = 0, err = 0, msgtype = RTM_MISS; + int needlock; KASSERT((fibnum < rt_numfibs), ("rtalloc1_fib: bad fibnum")); if (dst->sa_family != AF_INET) /* Only INET supports > 1 fib now */ @@ -283,59 +285,89 @@ rtalloc1_fib(struct sockaddr *dst, int r V_rtstat.rts_unreach++; goto miss2; } - RADIX_NODE_HEAD_LOCK(rnh); - if ((rn = rnh->rnh_matchaddr(dst, rnh)) && - (rn->rn_flags & RNF_ROOT) == 0) { + needlock = !(ignflags & RTF_RNH_LOCKED); +retry: + if (needlock) + RADIX_NODE_HEAD_RLOCK(rnh); + rn = rnh->rnh_matchaddr(dst, rnh); + if (rn && ((rn->rn_flags & RNF_ROOT) == 0)) { + + newrt = rt = RNTORT(rn); + nflags = rt->rt_flags & ~ignflags; + if (report && (nflags & RTF_CLONING)) { + if (needlock && !RADIX_NODE_HEAD_LOCK_TRY_UPGRADE(rnh)) { + RADIX_NODE_HEAD_RUNLOCK(rnh); + RADIX_NODE_HEAD_LOCK(rnh); + /* + * lookup again to make sure it wasn't changed + */ + rn = rnh->rnh_matchaddr(dst, rnh); + if (!(rn && ((rn->rn_flags & RNF_ROOT) == 0))) { + RADIX_NODE_HEAD_UNLOCK(rnh); + needresolve = 0; + log(LOG_INFO, "retrying route lookup ...\n"); + goto retry; + } + } + needresolve = 1; + } else { + RT_LOCK(newrt); + RT_ADDREF(newrt); + RADIX_NODE_HEAD_RUNLOCK(rnh); + goto done; + } + } + + if (!needresolve && needlock) + RADIX_NODE_HEAD_RUNLOCK(rnh); + + + if (needresolve) { + /* * If we find it and it's not the root node, then * get a reference on the rtentry associated. */ - newrt = rt = RNTORT(rn); - nflags = rt->rt_flags & ~ignflags; - if (report && (nflags & RTF_CLONING)) { /* * We are apparently adding (report = 0 in delete). * If it requires that it be cloned, do so. * (This implies it wasn't a HOST route.) */ - err = rtrequest_fib(RTM_RESOLVE, dst, NULL, - NULL, 0, &newrt, fibnum); - if (err) { - /* - * If the cloning didn't succeed, maybe - * what we have will do. Return that. - */ - newrt = rt; /* existing route */ - RT_LOCK(newrt); - RT_ADDREF(newrt); - goto miss; - } - KASSERT(newrt, ("no route and no error")); - RT_LOCK(newrt); - if (newrt->rt_flags & RTF_XRESOLVE) { - /* - * If the new route specifies it be - * externally resolved, then go do that. - */ - msgtype = RTM_RESOLVE; - goto miss; - } - /* Inform listeners of the new route. */ - bzero(&info, sizeof(info)); - info.rti_info[RTAX_DST] = rt_key(newrt); - info.rti_info[RTAX_NETMASK] = rt_mask(newrt); - info.rti_info[RTAX_GATEWAY] = newrt->rt_gateway; - if (newrt->rt_ifp != NULL) { - info.rti_info[RTAX_IFP] = - newrt->rt_ifp->if_addr->ifa_addr; - info.rti_info[RTAX_IFA] = newrt->rt_ifa->ifa_addr; - } - rt_missmsg(RTM_ADD, &info, newrt->rt_flags, 0); - } else { + err = rtrequest_fib(RTM_RESOLVE, dst, NULL, + NULL, RTF_RNH_LOCKED, &newrt, fibnum); + if (err) { + /* + * If the cloning didn't succeed, maybe + * what we have will do. Return that. + */ + newrt = rt; /* existing route */ RT_LOCK(newrt); RT_ADDREF(newrt); + goto miss; } - RADIX_NODE_HEAD_UNLOCK(rnh); + KASSERT(newrt, ("no route and no error")); + RT_LOCK(newrt); + if (newrt->rt_flags & RTF_XRESOLVE) { + /* + * If the new route specifies it be + * externally resolved, then go do that. + */ + msgtype = RTM_RESOLVE; + goto miss; + } + /* Inform listeners of the new route. */ + bzero(&info, sizeof(info)); + info.rti_info[RTAX_DST] = rt_key(newrt); + info.rti_info[RTAX_NETMASK] = rt_mask(newrt); + info.rti_info[RTAX_GATEWAY] = newrt->rt_gateway; + if (newrt->rt_ifp != NULL) { + info.rti_info[RTAX_IFP] = + newrt->rt_ifp->if_addr->ifa_addr; + info.rti_info[RTAX_IFA] = newrt->rt_ifa->ifa_addr; + } + rt_missmsg(RTM_ADD, &info, newrt->rt_flags, 0); + if (needlock) + RADIX_NODE_HEAD_UNLOCK(rnh); } else { /* * Either we hit the root or couldn't find any match, @@ -344,7 +376,8 @@ rtalloc1_fib(struct sockaddr *dst, int r */ V_rtstat.rts_unreach++; miss: - RADIX_NODE_HEAD_UNLOCK(rnh); + if (needlock && needresolve) + RADIX_NODE_HEAD_UNLOCK(rnh); miss2: if (report) { /* * If required, report the failure to the supervising @@ -356,6 +389,7 @@ rtalloc1_fib(struct sockaddr *dst, int r rt_missmsg(msgtype, &info, 0, err); } } +done: if (newrt) RT_LOCK_ASSERT(newrt); return (newrt); @@ -475,6 +509,8 @@ rtredirect_fib(struct sockaddr *dst, short *stat = NULL; struct rt_addrinfo info; struct ifaddr *ifa; + struct radix_node_head *rnh = + V_rt_tables[rt->rt_fibnum][dst->sa_family]; /* verify the gateway is directly reachable */ if ((ifa = ifa_ifwithnet(gateway)) == NULL) { @@ -524,14 +560,17 @@ rtredirect_fib(struct sockaddr *dst, info.rti_info[RTAX_NETMASK] = netmask; info.rti_ifa = ifa; info.rti_flags = flags; + if (rt0) + RT_UNLOCK(rt0); /* drop lock to avoid LOR with RNH */ error = rtrequest1_fib(RTM_ADD, &info, &rt, fibnum); - if (rt != NULL) { + if (rt) { RT_LOCK(rt); - EVENTHANDLER_INVOKE(route_redirect_event, rt0, rt, dst); + if (rt0) + EVENTHANDLER_INVOKE(route_redirect_event, rt0, rt, dst); flags = rt->rt_flags; } if (rt0) - RTFREE_LOCKED(rt0); + RTFREE(rt0); stat = &V_rtstat.rts_dynamic; } else { @@ -547,8 +586,12 @@ rtredirect_fib(struct sockaddr *dst, /* * add the key and gateway (in one malloc'd chunk). */ + RT_UNLOCK(rt); + RADIX_NODE_HEAD_LOCK(rnh); + RT_LOCK(rt); rt_setgate(rt, rt_key(rt), gateway); - gwrt = rtalloc1(gateway, 1, 0); + gwrt = rtalloc1(gateway, 1, RTF_RNH_LOCKED); + RADIX_NODE_HEAD_UNLOCK(rnh); EVENTHANDLER_INVOKE(route_redirect_event, rt, gwrt, dst); RTFREE_LOCKED(gwrt); } @@ -782,7 +825,9 @@ rtexpunge(struct rtentry *rt) struct ifaddr *ifa; int error = 0; + rnh = V_rt_tables[rt->rt_fibnum][rt_key(rt)->sa_family]; RT_LOCK_ASSERT(rt); + RADIX_NODE_HEAD_LOCK_ASSERT(rnh); #if 0 /* * We cannot assume anything about the reference count @@ -798,8 +843,6 @@ rtexpunge(struct rtentry *rt) if (rnh == NULL) return (EAFNOSUPPORT); - RADIX_NODE_HEAD_LOCK(rnh); - /* * Remove the item from the tree; it should be there, * but when callers invoke us blindly it may not (sigh). @@ -853,7 +896,6 @@ rtexpunge(struct rtentry *rt) */ V_rttrash++; bad: - RADIX_NODE_HEAD_UNLOCK(rnh); return (error); } @@ -868,7 +910,7 @@ rtrequest1_fib(int req, struct rt_addrin u_int fibnum) { INIT_VNET_NET(curvnet); - int error = 0; + int error = 0, needlock = 0; register struct rtentry *rt; register struct radix_node *rn; register struct radix_node_head *rnh; @@ -885,7 +927,10 @@ rtrequest1_fib(int req, struct rt_addrin rnh = V_rt_tables[fibnum][dst->sa_family]; if (rnh == NULL) return (EAFNOSUPPORT); - RADIX_NODE_HEAD_LOCK(rnh); + needlock = ((flags & RTF_RNH_LOCKED) == 0); + flags &= ~RTF_RNH_LOCKED; + if (needlock) + RADIX_NODE_HEAD_LOCK(rnh); /* * If we are adding a host route then we don't want to put * a netmask in the tree, nor do we want to clone it. @@ -1198,7 +1243,8 @@ deldone: error = EOPNOTSUPP; } bad: - RADIX_NODE_HEAD_UNLOCK(rnh); + if (needlock) + RADIX_NODE_HEAD_UNLOCK(rnh); return (error); #undef senderr } @@ -1305,7 +1351,8 @@ rt_setgate(struct rtentry *rt, struct so again: RT_LOCK_ASSERT(rt); - + RADIX_NODE_HEAD_LOCK_ASSERT(rnh); + /* * A host route with the destination equal to the gateway * will interfere with keeping LLINFO in the routing @@ -1331,7 +1378,7 @@ again: struct rtentry *gwrt; RT_UNLOCK(rt); /* XXX workaround LOR */ - gwrt = rtalloc1_fib(gate, 1, 0, rt->rt_fibnum); + gwrt = rtalloc1_fib(gate, 1, RTF_RNH_LOCKED, rt->rt_fibnum); if (gwrt == rt) { RT_REMREF(rt); return (EADDRINUSE); /* failure */ @@ -1402,12 +1449,8 @@ again: arg.rnh = rnh; arg.rt0 = rt; - RT_UNLOCK(rt); /* XXX workaround LOR */ - RADIX_NODE_HEAD_LOCK(rnh); - RT_LOCK(rt); rnh->rnh_walktree_from(rnh, rt_key(rt), rt_mask(rt), rt_fixchange, &arg); - RADIX_NODE_HEAD_UNLOCK(rnh); } return 0; Modified: user/kmacy/HEAD_fast_multi_xmit/sys/net/route.h ============================================================================== --- user/kmacy/HEAD_fast_multi_xmit/sys/net/route.h Sat Dec 6 09:37:54 2008 (r185690) +++ user/kmacy/HEAD_fast_multi_xmit/sys/net/route.h Sat Dec 6 09:38:41 2008 (r185691) @@ -197,6 +197,7 @@ struct ortentry { #define RTF_BROADCAST 0x400000 /* route represents a bcast address */ #define RTF_MULTICAST 0x800000 /* route represents a mcast address */ /* 0x1000000 and up unassigned */ +#define RTF_RNH_LOCKED 0x40000000 #define RTF_DESTEN_VALID 0x80000000 /* rtentry_info L2 addr is valid */ /* Mask of RTF flags that are allowed to be modified by RTM_CHANGE. */ Modified: user/kmacy/HEAD_fast_multi_xmit/sys/net/rtsock.c ============================================================================== --- user/kmacy/HEAD_fast_multi_xmit/sys/net/rtsock.c Sat Dec 6 09:37:54 2008 (r185690) +++ user/kmacy/HEAD_fast_multi_xmit/sys/net/rtsock.c Sat Dec 6 09:38:41 2008 (r185691) @@ -43,6 +43,7 @@ #include #include #include +#include #include #include #include @@ -554,11 +555,11 @@ route_output(struct mbuf *m, struct sock rnh = V_rt_tables[so->so_fibnum][info.rti_info[RTAX_DST]->sa_family]; if (rnh == NULL) senderr(EAFNOSUPPORT); - RADIX_NODE_HEAD_LOCK(rnh); + RADIX_NODE_HEAD_RLOCK(rnh); rt = (struct rtentry *) rnh->rnh_lookup(info.rti_info[RTAX_DST], info.rti_info[RTAX_NETMASK], rnh); if (rt == NULL) { /* XXX looks bogus */ - RADIX_NODE_HEAD_UNLOCK(rnh); + RADIX_NODE_HEAD_RUNLOCK(rnh); senderr(ESRCH); } #ifdef RADIX_MPATH @@ -574,14 +575,14 @@ route_output(struct mbuf *m, struct sock (rtm->rtm_type != RTM_GET || info.rti_info[RTAX_GATEWAY])) { rt = rt_mpath_matchgate(rt, info.rti_info[RTAX_GATEWAY]); if (!rt) { - RADIX_NODE_HEAD_UNLOCK(rnh); + RADIX_NODE_HEAD_RUNLOCK(rnh); senderr(ESRCH); } } #endif RT_LOCK(rt); RT_ADDREF(rt); - RADIX_NODE_HEAD_UNLOCK(rnh); + RADIX_NODE_HEAD_RUNLOCK(rnh); /* * Fix for PR: 82974 Modified: user/kmacy/HEAD_fast_multi_xmit/sys/netinet6/in6_rmx.c ============================================================================== --- user/kmacy/HEAD_fast_multi_xmit/sys/netinet6/in6_rmx.c Sat Dec 6 09:37:54 2008 (r185690) +++ user/kmacy/HEAD_fast_multi_xmit/sys/netinet6/in6_rmx.c Sat Dec 6 09:38:41 2008 (r185691) @@ -83,6 +83,7 @@ __FBSDID("$FreeBSD$"); #include #include #include +#include #include #include #include Modified: user/kmacy/HEAD_fast_multi_xmit/sys/netinet6/nd6_rtr.c ============================================================================== --- user/kmacy/HEAD_fast_multi_xmit/sys/netinet6/nd6_rtr.c Sat Dec 6 09:37:54 2008 (r185690) +++ user/kmacy/HEAD_fast_multi_xmit/sys/netinet6/nd6_rtr.c Sat Dec 6 09:38:41 2008 (r185691) @@ -44,6 +44,7 @@ __FBSDID("$FreeBSD$"); #include #include #include +#include #include #include #include