Date: Tue, 31 Dec 2013 17:11:59 -0800 From: Adrian Chadd <adrian@freebsd.org> To: FreeBSD Net <freebsd-net@freebsd.org> Subject: Re: ipv6 lock contention with parallel socket io Message-ID: <CAJ-Vmo=EbDQ8gBJA-zVAWo13Yo5b9Nir=yujJLHbSUrY7SQgHg@mail.gmail.com> In-Reply-To: <CAJ-VmonNDMZpCrojXRP92cUxzMKox_QX_5tBXjGM6kjei=oqjw@mail.gmail.com> References: <CAJ-VmoknC7TszvhonHgFq6mJNpgUmvXAjAH3uK9TMU0uMzcocw@mail.gmail.com> <CAJ-VmonNDMZpCrojXRP92cUxzMKox_QX_5tBXjGM6kjei=oqjw@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
btw - just committed this to -HEAD. I'll MFC it to stable/10 in a few days if noone raises any objections. Thanks, -a On 31 December 2013 09:53, Adrian Chadd <adrian@freebsd.org> wrote: > On 30 December 2013 23:35, Adrian Chadd <adrian@freebsd.org> wrote: >> Hi, >> >> I've noticed a bunch of lock contention occurs when doing highly >> parallel (> 4096 sockets) TCP IPv6 traffic. >> >> The contention is here: >> > [snip] > >> >> .. it's the IF_ADATA lock surrounding the lla_lookup() calls. >> >> Now: >> >> * is there any reason this isn't an rmlock? >> * the instance early on in nd6_output_lle() isn't taking the read >> lock, it's taking the full lock. Is there any reason for this? >> >> I don't have much experience or time to spend on optimising the ipv6 >> code to scale better but this seems like one of those things that'll >> bite us in the ass as the amount of ipv6 deployed increases. >> >> Does anyone have any ideas/suggestions on how we could improve things? > > This improves things quite a bit - from 1.9gbyte/sec @ 100% cpu usage > to 2.7gbyte/sec @ 85% CPU usage. It's not perfect - the lock > contention is still there - but it's much less of an issue now. > > Are there any issues with it? > > Index: sys/netinet6/nd6.c > =================================================================== > --- sys/netinet6/nd6.c (revision 259475) > +++ sys/netinet6/nd6.c (working copy) > @@ -1891,9 +1891,9 @@ > flags = ((m != NULL) || (lle != NULL)) ? LLE_EXCLUSIVE : 0; > if (ln == NULL) { > retry: > - IF_AFDATA_LOCK(ifp); > + IF_AFDATA_RLOCK(ifp); > ln = lla_lookup(LLTABLE6(ifp), flags, (struct sockaddr *)dst); > - IF_AFDATA_UNLOCK(ifp); > + IF_AFDATA_RUNLOCK(ifp); > if ((ln == NULL) && nd6_is_addr_neighbor(dst, ifp)) { > /* > * Since nd6_is_addr_neighbor() internally > calls nd6_lookup(), > > > Thanks, > > > > -a
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAJ-Vmo=EbDQ8gBJA-zVAWo13Yo5b9Nir=yujJLHbSUrY7SQgHg>